1 / 24

a generic test-system

a generic test-system. Pieter Koopman , Artem Alimarine, Jan Tretmans, Rinus Plasmeijer Nijmegen, NL. overview. testing: why what will be tested how do we test by an automatic system generic testing by a generic system automatic and systematic generation of test data

aimon
Download Presentation

a generic test-system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. a generic test-system Pieter Koopman, Artem Alimarine, Jan Tretmans, Rinus Plasmeijer Nijmegen, NL

  2. overview testing: • why • what will be tested • how do we test • by an automatic system • generic testing by a generic system • automatic and systematic generation of test data • research question: can we make such a system • conclusions / related & future work

  3. why testing • even in functional programming we make errors • even errors that the type system does not spot • to improve quality and confidence the software • proving is often too much work, or not feasible • manual testing is • much work (up to 50% of project cost) • dull • to be repeated after changes  make an automatic system • encourages writing specifications • makes testing easier, faster and more reliable

  4. what will be tested • there is a rich palette of quality aspects • suitability: validating • obeying the specification: functional testing • efficiency • ... we restrict ourselves to:obeying the specification, functional testing • specification can be: • executable specification inefficient but obviously correct • reference implementation older version • relation between input and output • ...

  5. specification • specification by functions in Clean • prop_DeMorgan :: Bool Bool -> Boolprop_DeMorgan x y = x && y == not (not x || not y) semantics:x,y•x&&y==not(notx||not y) • prop_DropDrop :: Int Int [Int] -> Boolprop_DropDrop m n xs = drop n (drop m xs) == drop (n+m) xs • prop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==>let r = sqrt x in r*r == x • monomorph in order to generate test data • for polymorphic functions this tests all types • for overloaded functions the instance does matter

  6. how: basic idea • generate test data, finite and   • check property for all these values • test prop | and [ evaluateprop x\\x<- generate] = "Passed" = "Counter-example found" • requirements • onegenerate for all types • information about the counter-example • information about test data used • what test data should be generated ?

  7. what test data? • random ? (as in QuickCheck by Claessen & Hughes)+ easy to implement+ easy to use+ works pretty well for small programs- are all relevant cases covered?- duplicates (which are useless) • systematic !+ coverage of standard cases (guaranteed) Int: 0, 1, ... List Int: [ ], [ 0 ], [ 1 ], ...+ no duplicates+ more confidence, better results- harder to implement

  8. systematic generation of test data what do we want? • finite types:: Bool = True | False:: Colour = Red | Yellow | Blue • all values can and should be tested • very large types (always basic types)Int, Real, Char, String, ... • common border values (like 0, 1) and random values • recursive typesList a = Nil | Cons a (List a)Tree a = Tip | Node a (Tree a) (Tree a) • always handled by recursive functions • systematic generation from small to large

  9. systematic generation 2 • basic types (Int, Real, ...) are handled separately • for all other types systematic generation from small to large is better than random • covers interesting cases • avoids duplicates • how to define one general generate function ? • one function for all types • no new instance for each new type • systematic generation • no duplicates • small to large • use generics !

  10. what are generics • uniform tree representation of types • conversion between actual type and tree representation by system • tree representation can be manipulated in generic functions • we use standard generics • See Hinze, Alimarine, ... • enables us to define onegenerate function • systematic generation of generic trees • automatic conversion of trees to needed types

  11. Red° Yellow° Blue° tree representation of all colours generic trees • :: UNIT= UNIT // tip of tree:: EITHER a b= LEFT a | RIGHT b // choice:: PAIRa b= PAIR a b // product type • example::: Colour = Red | Yellow | Blue • generic representation of this type:: Colour° = EITHER UNIT (EITHER UNIT UNIT) • generic representation of constructorsRed° = LEFT UNITYellow° = RIGHT (LEFT UNIT)Blue° = RIGHT (RIGHT UNIT)

  12. tree of all possible values of type [ Colour° ]° Cons [] Cons Red [] Yellow Blue ... ... generic trees 2 representation of [] (which is Nil): • LEFT UNIT note: same representation as Red! representation of [Red] ( Cons Red Nil): • RIGHT (PAIR (LEFT UNIT) (LEFT UNIT))

  13. F A B toGen fromGen A° B° F° generic functions • Basic idea: • instead of a function F :: A -> B for each A and B • we define a generic function F° • we implement F as GenToB o F° o AToGen • transformations are generated by the compiler

  14. eq T T Bool eq T° T° Bool Example a generic equality • generic eq a :: a a -> Bool • instances for generic type: eq{|UNIT|}UNITUNIT = True • eq{|PAIR|} eqx eqy (PAIR x1 y1) (PAIR x2 y2) = eqx x1 x2 && eqy y1 y2 • eq{|EITHER|} eql eqr (LEFT x) (LEFT y) = eql x yeq{|EITHER|} eql eqr (RIGHT x) (RIGHT y) = eqr x yeq{|EITHER|} eql eqr e1 e2 = False • eq{|Int|} n m = n == m // similar for other basic types • for each type T we either use the generic version:derive eq T • or define an own instance:eq{|T|} t1 t2 = ...

  15. function generate • using generics we can define onegenerate function algorithm: • systematic generation of generic trees • automatic conversion to desired type

  16. systematic generation of generic trees • preorder traversal • remember current tree • first all subtrees starting with LEFT • then all subtrees starting with RIGHT • works fine for Colour, [ Colour], ... • problems if left generic subtree is infinite • Tree a = Tip | Node a (Tree a) (Tree a)using :: T = CTree Tgenerates: Tip, NodeCTip Tip, Node C (Node C Tip Tip) Tip, Node C (Node C (Node C Tip Tip) Tip) Tip, ...never generates Node C Tip (Node C Tip Tip),... • [[T]]generates: [ ], [[ ]], [[C]], [[C, C]], [[C, C, C]],...never generates [[ ], [ ]], [[ ], [C]], [[C], [ ]],...

  17. generation of generic trees 2 • breadth first traversal • order trees with respect to number of LEFT/RIGHT nodes, or constructors • yield trees with increasing depth • within same depth: left to right • problems • not easy to generate trees with increasing depth efficiently • not every tree represents a valid data value • adding a constructor requires adding its arguments • large administration needed

  18. generation of generic trees 3 • preorder does not work with left recursion • breath first is nasty • use mixed approach • remember generated branches of tree • extend tree for next test-data item • at each EITHER node choose randomly between extending LEFT or RIGHT branch • Properties: • systematic • no duplicates • interesting cases occur soon (with high probability) • also bigger values earlier in list of test-data

  19. testing • administrate arguments used in a record • :: Result = { ok :: Maybe Bool, args :: [String] } • define class Testable: • class Testable awhere evaluate :: a RandomStream Result -> [Result] • instance Testable Boolwhere evaluate b rs result = [{result & ok = Just b}] • instance Testable (a->b) | Testable b & TestArg a where evaluate f rs result = let (rs,rs2) = split rsin forAll rs2 f result (generate rs)forAll rs f r list= diagonal [apply (genRand seed) f a r \\ a<-list & seed <- rs ]apply rs f a r = evaluate (f a) rs {r & args = [show a:r.args]}use generic functions generate and show

  20. conditional tests • prop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==>letr = sqrt x inr*r == x • implementation • :: Property = Prop (RandomStream Result -> [Result]) • (==>) infixr 0 :: Bool p -> Property | Testable p(==>) b p • | b= Prop (evaluate p)// continue testing= Prop (\rs r = [r])// stop testing, dummy result • instance Testable Property • where evaluate (Prop p) rs result = p rs result

  21. example results • prop_Sqrt :: Real -> Propertyprop_Sqrt x = x>=0.0 ==>letr = sqrt x inr*r == x • counter-example found after 2 tests: .1191576 • prop_DropDrop :: Int Int [Int] -> Boolprop_DropDrop m n xs = drop n (drop m xs) == drop (n+m) xs • counter-example found after 13 tests: -1 1 [0,0] • prop_RevRev :: [Int] -> Propertyprop_RevRev xs = classify(isEmpty xs) xs(reverse (reverse xs)==xs ) • Passed 1000 tests[]: 1 (0.1%) • prop_DeMorgan :: Bool Bool -> Boolprop_DeMorgan x y = x && y == not (not x || not y) • counter-example found after 3 tests: False True • with correction specification: Passed4 tests

  22. conclusions • a test system is useful • writing specifications is encouraged • testing improves quality and confidence • a generic test system is useful • property is arbitrary Clean function • system applies predicate to test-data • system generates test-data automatically • use generics to generate, show and compare values • systematic generation of test data • no duplicated tests • covers interesting cases • stops if all cases are tested

  23. related work • for FPL: QuickCheck • advantages of our generic approach • one generate function for all types • no user defined instances needed • generate • show • equal • systematic generation of data • no duplicates • covers interesting cases • stops when all cases are tested

  24. future work • handling  • generating better functions • prop_Map :: (Int->Int) (Int->Int) [Int] -> Boolprop_Map f g xs = map f (map g xs) == map (f o g) xs • easier user control over test-data • better info about test-data used • case studies • GUI • testing application specified in Clean but written in some other programming language • integration with proof-system • ...

More Related