Demonstrating and testing the bml compliance of bml realizers
Download
1 / 26

Demonstrating and Testing the BML Compliance of BML Realizers - PowerPoint PPT Presentation


  • 65 Views
  • Uploaded on

Demonstrating and Testing the BML Compliance of BML Realizers. Herwin van Welbergen, Yuyu Xu , Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro. What is this talk about. Testing the adherence to the BML standard by realizers

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Demonstrating and Testing the BML Compliance of BML Realizers' - taini


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Demonstrating and testing the bml compliance of bml realizers

Demonstrating and Testing the BML Compliance of BML Realizers

Herwin van Welbergen, Yuyu Xu,

Marcus Thiebaux, Wei-Wen Feng,

Jingqiao Fu, Dennis Reidsma, Ari Shapiro


What is this talk about
What is this talk about

  • Testing the adherence to the BML standard by realizers

  • (and, by this, also testing for weaknesses in the standard)

  • Better software engineering practices for virtual humans

    • Daily automated testing (regression, acceptance, …)


Bml realizer interface saiba framework
BML Realizer Interface:SAIBA Framework



Bml realizer interface feedback comes out
BML Realizer Interface: Feedback comes out

  • bml1 starts (globaltime=10)

  • gaze1 starts (localtime=0)

  • gaze1 ready (localtime=1)

  • speech1 start (localtime=1)

  • speech1 end (locatime=10)

  • bml1 ends (globaltime=20)


Why automatic testing of realizers
Why automatic testing of realizers?

  • BML Realizers are complex software components

  • They form the backbone of virtual human applications by research groups

  • Yet their testing has so far been limited to time consuming manual inspection of the execution of BML scripts


Why automatic testing of realizers1
Why automatic testing of realizers?

  • Automatic tests are automatic

    • No wasted developer time

  • Automatic tests can be run often

    • Early error detection


How can we test a realizer
How can we test a realizer?

  • A realizer provides standardized interface

    • BML goes in

    • Feedback comes out

  • Several realizers implement this interface

  • We can make automatic tests for this interface

  • These tests can be used for all realizers implementing the interface


What can be tested
What can be tested?

  • Message flow and behavior execution

  • Adherence to time constraints

  • Error handling

  • Acceptance testing


Typical test case
Typical test case

  • Send BML to a realizer, capture all feedback

  • Wait until the realizer has executed the BML

  • Verify assertions on the received feedback


Example test case
Example test case

<bml id="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

Whatcan we assertabout the expected feedback?

- All default sync points remain in order forallbehaviors

- The headnodwill start within |ε| seconds of the start of “what”

- Speech1 starts within |ε| seconds of t=6

- Therewillbe no exceptions or warnings

- The block will end


Example test case1
Example test case

<bmlid="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

@Test public void testSpeechNodTimedToSync() {

realizerPort.performBML(readTestFile("testspeech_nodtimedtosync.xml"));

waitForBMLEndFeedback("bml1");

assertSyncsInOrder("bml1", "speech1", "start", "ready", "stroke_start",

"stroke", "s1", "stroke_end", "relax", "end");

assertAllBMLSyncsInBMLOrder("bml1", "nod1");

assertBlockStartAndStopFeedbacks("bml1");

assertRelativeSyncTime("bml1", "speech1", "start", 6);

assertLinkedSyncs("bml1", "speech1", "s1", "bml1", "nod1", "start");

assertNoExceptions();

assertNoWarnings();

}


Realizertester
RealizerTester

  • A generic testing framework that can test all BML Realizers

  • Provides a set of tests + assertions

  • And functionality to easily author new tests

    • Utility methods (e.g. waitForBMLEndFeedback)

    • Custom assertions (e.g. assertLinkedSyncs)

  • Code and tests released on SF, together with a nice set of example movies to showcase various realizers


Testing elckerlyc
Testing Elckerlyc

  • Using RealizerTester

    • Additional tests for Elckerlyc specific functionality

    • Regression testing

    • Acceptance testing

  • Running all tests takes some time (~3 min.)

    • This might discourage frequent testing by developers

    • => Run tests automatically on a continuous integration server

      • Notify developers that commit a build that fails tests

      • Keep track of test result history


Testing elckerlyc1
Testing Elckerlyc

  • RealizerTester does not help in identifying the exact location of errors

    • It tests a Realizer as a black box

    • Elckerlyc uses additional white box testing at a smaller granularity to identify the exact location of errors

      • >1000 unit + mid range tests, running in <10s

      • RealizerTester helps in identifying locations that require more unit testing

  • Regular visual inspection is still valuable!


Testing in smartbody
Testing in SmartBody

  • Visual regression testing

    • Record a baseline

    • Does it (approximately) look like the baseline?

    • Automatically check difference with baseline

    • If different

      • Bug: fix

      • New functionality: create new baseline


Testing in smartbody example
Testing in SmartBody: example


Testing in smartbody1
Testing in SmartBody

  • Visual regression testing:

    • checks whether the correct motion is generated rather than whether the correct signals are sent

    • is Realizer and character dependent

    • does not provide acceptance testing

  • RealizerTester is complementary to visual regression testing


Testing smartbody using realizertester
Testing SmartBody using RealizerTester

  • Dealing with BML versions

  • Took 1 day to implement

  • Found someunclarities in BML (e.g. feedback ordering) nowaddressed in 1.0

  • Found some minor implementation issues in SmartBody

  • Did not find any interpretation differences in the constraintsatisfaction





Observations
Observations

  • Designing test cases + assertions

    • Highlighted several cases in which the current BML specification lacked detail or was unclear

  • Setting up a video corpus

    • Highlighted some expressivity issues in the BML standard

    • Shows that several behaviors are executed in a semantically equivalent manner

    • Created a healthy competition between the developers

    • Motivated developers to move to better BML compliance


Conclusions
Conclusions

  • The modularity proposed by SAIBA enables the reuse of testing functionality

  • Designing a generic testing framework and a shared video repository helps move the standard forward

  • BMLRealizerTester provides a starting point for a BML compliance test suite


Further work
Furtherwork

  • XML format to author tests using BML and assertions expressed?

  • Add more realizers (PLEASE CONTACT US!)

    • Full BML compliance is not required

    • Support for feedback is required

  • Includenewestversion of BML standard discussed last Wednesday

  • Continue extendingourrepository of niceexamplemovies


Thanks for your attention
Thanks for your attention

http://realizertester.sourceforge.net/

  • contact: [email protected]


ad