demonstrating and testing the bml compliance of bml realizers
Download
Skip this Video
Download Presentation
Demonstrating and Testing the BML Compliance of BML Realizers

Loading in 2 Seconds...

play fullscreen
1 / 26

Demonstrating and Testing the BML Compliance of BML Realizers - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Demonstrating and Testing the BML Compliance of BML Realizers. Herwin van Welbergen, Yuyu Xu , Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro. What is this talk about. Testing the adherence to the BML standard by realizers

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Demonstrating and Testing the BML Compliance of BML Realizers' - taini


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
demonstrating and testing the bml compliance of bml realizers

Demonstrating and Testing the BML Compliance of BML Realizers

Herwin van Welbergen, Yuyu Xu,

Marcus Thiebaux, Wei-Wen Feng,

Jingqiao Fu, Dennis Reidsma, Ari Shapiro

what is this talk about
What is this talk about
  • Testing the adherence to the BML standard by realizers
  • (and, by this, also testing for weaknesses in the standard)
  • Better software engineering practices for virtual humans
    • Daily automated testing (regression, acceptance, …)
bml realizer interface feedback comes out
BML Realizer Interface: Feedback comes out
  • bml1 starts (globaltime=10)
  • gaze1 starts (localtime=0)
  • gaze1 ready (localtime=1)
  • speech1 start (localtime=1)
  • speech1 end (locatime=10)
  • bml1 ends (globaltime=20)
why automatic testing of realizers
Why automatic testing of realizers?
  • BML Realizers are complex software components
  • They form the backbone of virtual human applications by research groups
  • Yet their testing has so far been limited to time consuming manual inspection of the execution of BML scripts
why automatic testing of realizers1
Why automatic testing of realizers?
  • Automatic tests are automatic
    • No wasted developer time
  • Automatic tests can be run often
    • Early error detection
how can we test a realizer
How can we test a realizer?
  • A realizer provides standardized interface
    • BML goes in
    • Feedback comes out
  • Several realizers implement this interface
  • We can make automatic tests for this interface
  • These tests can be used for all realizers implementing the interface
what can be tested
What can be tested?
  • Message flow and behavior execution
  • Adherence to time constraints
  • Error handling
  • Acceptance testing
typical test case
Typical test case
  • Send BML to a realizer, capture all feedback
  • Wait until the realizer has executed the BML
  • Verify assertions on the received feedback
example test case
Example test case

<bml id="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

Whatcan we assertabout the expected feedback?

- All default sync points remain in order forallbehaviors

- The headnodwill start within |ε| seconds of the start of “what”

- Speech1 starts within |ε| seconds of t=6

- Therewillbe no exceptions or warnings

- The block will end

example test case1
Example test case

<bmlid="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

@Test public void testSpeechNodTimedToSync() {

realizerPort.performBML(readTestFile("testspeech_nodtimedtosync.xml"));

waitForBMLEndFeedback("bml1");

assertSyncsInOrder("bml1", "speech1", "start", "ready", "stroke_start",

"stroke", "s1", "stroke_end", "relax", "end");

assertAllBMLSyncsInBMLOrder("bml1", "nod1");

assertBlockStartAndStopFeedbacks("bml1");

assertRelativeSyncTime("bml1", "speech1", "start", 6);

assertLinkedSyncs("bml1", "speech1", "s1", "bml1", "nod1", "start");

assertNoExceptions();

assertNoWarnings();

}

realizertester
RealizerTester
  • A generic testing framework that can test all BML Realizers
  • Provides a set of tests + assertions
  • And functionality to easily author new tests
    • Utility methods (e.g. waitForBMLEndFeedback)
    • Custom assertions (e.g. assertLinkedSyncs)
  • Code and tests released on SF, together with a nice set of example movies to showcase various realizers
testing elckerlyc
Testing Elckerlyc
  • Using RealizerTester
    • Additional tests for Elckerlyc specific functionality
    • Regression testing
    • Acceptance testing
  • Running all tests takes some time (~3 min.)
    • This might discourage frequent testing by developers
    • => Run tests automatically on a continuous integration server
      • Notify developers that commit a build that fails tests
      • Keep track of test result history
testing elckerlyc1
Testing Elckerlyc
  • RealizerTester does not help in identifying the exact location of errors
    • It tests a Realizer as a black box
    • Elckerlyc uses additional white box testing at a smaller granularity to identify the exact location of errors
      • >1000 unit + mid range tests, running in <10s
      • RealizerTester helps in identifying locations that require more unit testing
  • Regular visual inspection is still valuable!
testing in smartbody
Testing in SmartBody
  • Visual regression testing
    • Record a baseline
    • Does it (approximately) look like the baseline?
    • Automatically check difference with baseline
    • If different
      • Bug: fix
      • New functionality: create new baseline
testing in smartbody1
Testing in SmartBody
  • Visual regression testing:
    • checks whether the correct motion is generated rather than whether the correct signals are sent
    • is Realizer and character dependent
    • does not provide acceptance testing
  • RealizerTester is complementary to visual regression testing
testing smartbody using realizertester
Testing SmartBody using RealizerTester
  • Dealing with BML versions
  • Took 1 day to implement
  • Found someunclarities in BML (e.g. feedback ordering) nowaddressed in 1.0
  • Found some minor implementation issues in SmartBody
  • Did not find any interpretation differences in the constraintsatisfaction
observations
Observations
  • Designing test cases + assertions
    • Highlighted several cases in which the current BML specification lacked detail or was unclear
  • Setting up a video corpus
    • Highlighted some expressivity issues in the BML standard
    • Shows that several behaviors are executed in a semantically equivalent manner
    • Created a healthy competition between the developers
    • Motivated developers to move to better BML compliance
conclusions
Conclusions
  • The modularity proposed by SAIBA enables the reuse of testing functionality
  • Designing a generic testing framework and a shared video repository helps move the standard forward
  • BMLRealizerTester provides a starting point for a BML compliance test suite
further work
Furtherwork
  • XML format to author tests using BML and assertions expressed?
  • Add more realizers (PLEASE CONTACT US!)
    • Full BML compliance is not required
    • Support for feedback is required
  • Includenewestversion of BML standard discussed last Wednesday
  • Continue extendingourrepository of niceexamplemovies
thanks for your attention
Thanks for your attention

http://realizertester.sourceforge.net/

ad