Demonstrating and testing the bml compliance of bml realizers
This presentation is the property of its rightful owner.
Sponsored Links
1 / 26

Demonstrating and Testing the BML Compliance of BML Realizers PowerPoint PPT Presentation


  • 45 Views
  • Uploaded on
  • Presentation posted in: General

Demonstrating and Testing the BML Compliance of BML Realizers. Herwin van Welbergen, Yuyu Xu , Marcus Thiebaux, Wei-Wen Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro. What is this talk about. Testing the adherence to the BML standard by realizers

Download Presentation

Demonstrating and Testing the BML Compliance of BML Realizers

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Demonstrating and testing the bml compliance of bml realizers

Demonstrating and Testing the BML Compliance of BML Realizers

Herwin van Welbergen, Yuyu Xu,

Marcus Thiebaux, Wei-Wen Feng,

Jingqiao Fu, Dennis Reidsma, Ari Shapiro


What is this talk about

What is this talk about

  • Testing the adherence to the BML standard by realizers

  • (and, by this, also testing for weaknesses in the standard)

  • Better software engineering practices for virtual humans

    • Daily automated testing (regression, acceptance, …)


Bml realizer interface saiba framework

BML Realizer Interface:SAIBA Framework


Bml realizer interface bml goes in

BML Realizer Interface: BML goes in


Bml realizer interface feedback comes out

BML Realizer Interface: Feedback comes out

  • bml1 starts (globaltime=10)

  • gaze1 starts (localtime=0)

  • gaze1 ready (localtime=1)

  • speech1 start (localtime=1)

  • speech1 end (locatime=10)

  • bml1 ends (globaltime=20)


Why automatic testing of realizers

Why automatic testing of realizers?

  • BML Realizers are complex software components

  • They form the backbone of virtual human applications by research groups

  • Yet their testing has so far been limited to time consuming manual inspection of the execution of BML scripts


Why automatic testing of realizers1

Why automatic testing of realizers?

  • Automatic tests are automatic

    • No wasted developer time

  • Automatic tests can be run often

    • Early error detection


How can we test a realizer

How can we test a realizer?

  • A realizer provides standardized interface

    • BML goes in

    • Feedback comes out

  • Several realizers implement this interface

  • We can make automatic tests for this interface

  • These tests can be used for all realizers implementing the interface


What can be tested

What can be tested?

  • Message flow and behavior execution

  • Adherence to time constraints

  • Error handling

  • Acceptance testing


Typical test case

Typical test case

  • Send BML to a realizer, capture all feedback

  • Wait until the realizer has executed the BML

  • Verify assertions on the received feedback


Example test case

Example test case

<bml id="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

Whatcan we assertabout the expected feedback?

- All default sync points remain in order forallbehaviors

- The headnodwill start within |ε| seconds of the start of “what”

- Speech1 starts within |ε| seconds of t=6

- Therewillbe no exceptions or warnings

- The block will end


Example test case1

Example test case

<bmlid="bml1">

<speech id="speech1" start="6">

<text>Hey punk <sync id="s1" />what do ya want?</text>

</speech>

<head id="nod1" action="ROTATION" rotation="X" start="speech1:s1"/>

</bml>

@Test public void testSpeechNodTimedToSync() {

realizerPort.performBML(readTestFile("testspeech_nodtimedtosync.xml"));

waitForBMLEndFeedback("bml1");

assertSyncsInOrder("bml1", "speech1", "start", "ready", "stroke_start",

"stroke", "s1", "stroke_end", "relax", "end");

assertAllBMLSyncsInBMLOrder("bml1", "nod1");

assertBlockStartAndStopFeedbacks("bml1");

assertRelativeSyncTime("bml1", "speech1", "start", 6);

assertLinkedSyncs("bml1", "speech1", "s1", "bml1", "nod1", "start");

assertNoExceptions();

assertNoWarnings();

}


Realizertester

RealizerTester

  • A generic testing framework that can test all BML Realizers

  • Provides a set of tests + assertions

  • And functionality to easily author new tests

    • Utility methods (e.g. waitForBMLEndFeedback)

    • Custom assertions (e.g. assertLinkedSyncs)

  • Code and tests released on SF, together with a nice set of example movies to showcase various realizers


Testing elckerlyc

Testing Elckerlyc

  • Using RealizerTester

    • Additional tests for Elckerlyc specific functionality

    • Regression testing

    • Acceptance testing

  • Running all tests takes some time (~3 min.)

    • This might discourage frequent testing by developers

    • => Run tests automatically on a continuous integration server

      • Notify developers that commit a build that fails tests

      • Keep track of test result history


Testing elckerlyc1

Testing Elckerlyc

  • RealizerTester does not help in identifying the exact location of errors

    • It tests a Realizer as a black box

    • Elckerlyc uses additional white box testing at a smaller granularity to identify the exact location of errors

      • >1000 unit + mid range tests, running in <10s

      • RealizerTester helps in identifying locations that require more unit testing

  • Regular visual inspection is still valuable!


Testing in smartbody

Testing in SmartBody

  • Visual regression testing

    • Record a baseline

    • Does it (approximately) look like the baseline?

    • Automatically check difference with baseline

    • If different

      • Bug: fix

      • New functionality: create new baseline


Testing in smartbody example

Testing in SmartBody: example


Testing in smartbody1

Testing in SmartBody

  • Visual regression testing:

    • checks whether the correct motion is generated rather than whether the correct signals are sent

    • is Realizer and character dependent

    • does not provide acceptance testing

  • RealizerTester is complementary to visual regression testing


Testing smartbody using realizertester

Testing SmartBody using RealizerTester

  • Dealing with BML versions

  • Took 1 day to implement

  • Found someunclarities in BML (e.g. feedback ordering) nowaddressed in 1.0

  • Found some minor implementation issues in SmartBody

  • Did not find any interpretation differences in the constraintsatisfaction


Observations au 1

Observations: au 1


Observations gaze

Observations: gaze


Observations longer performance

Observations: longer performance


Observations

Observations

  • Designing test cases + assertions

    • Highlighted several cases in which the current BML specification lacked detail or was unclear

  • Setting up a video corpus

    • Highlighted some expressivity issues in the BML standard

    • Shows that several behaviors are executed in a semantically equivalent manner

    • Created a healthy competition between the developers

    • Motivated developers to move to better BML compliance


Conclusions

Conclusions

  • The modularity proposed by SAIBA enables the reuse of testing functionality

  • Designing a generic testing framework and a shared video repository helps move the standard forward

  • BMLRealizerTester provides a starting point for a BML compliance test suite


Further work

Furtherwork

  • XML format to author tests using BML and assertions expressed?

  • Add more realizers (PLEASE CONTACT US!)

    • Full BML compliance is not required

    • Support for feedback is required

  • Includenewestversion of BML standard discussed last Wednesday

  • Continue extendingourrepository of niceexamplemovies


Thanks for your attention

Thanks for your attention

http://realizertester.sourceforge.net/

  • contact: [email protected]


  • Login