1 / 31

Testing a Media Proxy with … QuickCheck

Testing a Media Proxy with … QuickCheck. Thomas Arts John Hughes Chalmers/ITU. Joakim Johansson Ulf Wiger Ericsson. QuickCheck: Properties not Test Cases. Write general properties of code instead of test cases Test in many, many randomly generated cases.

julio
Download Presentation

Testing a Media Proxy with … QuickCheck

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Testing a Media Proxy with … QuickCheck Thomas Arts John Hughes Chalmers/ITU Joakim Johansson Ulf Wiger Ericsson

  2. QuickCheck: Properties not Test Cases • Write general properties of code instead of test cases • Test in many, many randomly generated cases prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(Xs++Ys)== lists:reverse(Xs)++lists:reverse(Ys))).

  3. QuickCheck Testing • Simplification is extremely important—separates the signal from the noise! 3> eqc:quickcheck(test:prop_reverse()). ...........Failed! After 12 tests. [-3,2] [-3,1] Shrinking..........(10 times) [0] [1] false A random counter-example: Xs and Ys NEW: Automatic simplification of failing cases

  4. Why Did it Fail? • QuickCheck says: Xs=[0], Ys=[1] • reverse([0,1])==reverse([0])++reverse([1])? • Xs, Ys the wrong way round prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(Xs++Ys)== lists:reverse(Xs)++lists:reverse(Ys))).

  5. QuickCheck in a Nutshell • Features • Properties, not test cases • Controlled random generation • Automated simplification • Result • Compact, reusable test code • Specification checked against the code • Shortcut from property to bug

  6. A Bug! A Bug! Property First Development? • Code code code… quickcheck • Fix it… quickcheck • Fix it… quickcheck… quickcheck quickcheck quickcheck • Code some more…

  7. Ericsson’s Media Proxy Media Gateway Controller IP network, servers, etc etc Media stream

  8. How hard can it be? • All we need to do is open and close ”media pinholes” when the controller says… • Megaco H.248 protocol • ITU standard… 212 pages! • Ericsson Interwork Description… 183 pages! • Control software… 150KLOC • 20-30K Megaco

  9. Our Strategy QuickCheck Sends random command sequences Results make sense? Completely random commands are all just rejected—non-sensical! Poor test data

  10. Generating Sensible Messages

  11. Message Generators • Example: A media descriptor contains a list of streams • ASN.1 from the standard (simplified): MediaDescriptor ::= SEQUENCE { streams CHOICE { oneStream StreamParms, multiStream SEQUENCE OF StreamDescriptor } }

  12. QuickCheck Generator mediadescriptor(Streams) when Streams=/=[]-> {mediaDescriptor, #'MediaDescriptor'{ streams = case Streams of [{Id,Mode}] -> oneof([{oneStream,streamParms(Mode)}, {multiStream,[stream(Id,Mode)]}]); _ -> {multiStream, [stream(I,M) || {I,M}<-Streams]} end}}. stream(I,Mode) -> #'StreamDescriptor'{ streamID = I, streamParms = streamParms(Mode)}. Records generated by ASN.1 compiler Message construction Logic from the IWD QuickCheck

  13. Conditions in the IWD StreamParms ::= SEQUENCE { localControlDescriptor LocalControlDescriptor OPTIONAL, localDescriptor LocalRemoteDescriptor OPTIONAL, remoteDescriptor LocalRemoteDescriptor OPTIONAL, …, statisticsDescriptor StatisticsDescriptor OPTIONAL } Remote SDP in here

  14. Two Cases: With and Without Remote Media Included in this case streamParms(Mode) -> ?LET(RemoteMediaDefined, bool(), if RemoteMediaDefined -> #'StreamParms'{ localControlDescriptor = localControl(Mode), localDescriptor = localDescriptor(RemoteMediaDefined), remoteDescriptor = remoteDescriptor(RemoteMediaDefined)}; true -> …… end). Passed on to ensure an m-line is generated

  15. Generating Sensible Commands

  16. Megaco Commands Context Context

  17. Megaco Commands Context Context Context Termination Add

  18. Megaco Commands Context Context Add Context Termination Termination

  19. Megaco Commands Context Context Context Termination Termination Modify Stream Stream Uses termination ID

  20. Megaco Commands Context Context Subtract Context Termination Termination Stream Stream

  21. Megaco Commands Context Context Context Termination Subtract Stream Stream

  22. Megaco Commands Context Context

  23. Sensible Command Sequences? • Track the state of each test case • What calls in progress, which terminations in each call… • Simple to model in Erlang • State machine model • Preconditions say which commands are appropriate • Postconditions check command results • Next state function specifies how commands behave • QuickCheck library • generates valid sequences, tests, and shrinks them • (developed ”just in time”)

  24. Faults • Proxy was already well tested • 6 days of work, mostly writing generators • 5 faults found: • One in Megaco message encoding/decoding • Four command sequences crashed

  25. Add Mod Add Sub Add Mod Add Error-provoking Sequences Streams must have two ends Here today, gone tomorrow… …with differing numbers of streams

  26. Add Add Sub Add Sub Add Sub The Best Error! Shrinking reduced 160 commands to seven! No one in their right minds would test this! Due to data corruption here

  27. ”Most Likely Bug” Effect • Once a bug is found, almost every QuickCheck run finds the same one! • (or at least, shrinks to it) • Add bug preconditions to the model to avoid it and find the next one • Formulate a ”bug hypothesis” • Test the hypothesis! • Document it as a precondition

  28. Real bugs or not? • Most of these sequences are not sent by Ericsson’s Media Gateway Controller • Controller and Proxy were tested together! • But the Interwork Description doesn’t say so! • One of the IWD/the code was buggy!

  29. Using QuickCheck Earlier? • Testing an older version • Using the same generators and model • 9 errors found in 6 hours! • (2 TRs for this version) • Most of the work: writing bug preconditions

  30. QuickCheck is Moreish… • Follow-up projects under way • Testing non-Erlang systems (the big one!)

  31. Conclusions • QuickCheck + Erlang • Simple declarative models of the SUT • Concise, maintainable test code • Testing: a great application for FP • Performance irrelevant • No need to commit to Erlang in the product • A winning combination?

More Related