CS527 Topics in Software Engineering (Software Testing and Analysis)

1. CS527Topics in Software Engineering(Software Testing and Analysis) Darko Marinov August 30, 2011

2. Course Info • Everyone on the mailing list and “People” • Hopefully no more registration changes • Coursework • PROJECT [40%] • Already received some proposals • Presentation [20%] • Participation (reports and discussion) [20%] • Homework assignment(s) [20%] • Fair warning: this is NOT easy • Project requires that you explore some topic in great DEPTH

3. Schedule • First few lectures to help you select projects • Last week: Intro, Reading/Writing papers • Today: CHESS (find multithreaded bugs) • Sep 1: IMUnit (writing multithreaded tests) • Sep 6: CAPP & Setac (more concurrent testing) • Sep 8: iComment (analyzing source comments) • Your preferences? (I’ll also read HW0) • Software testing, model checking, static analysis • Your presentations from Oct 4 • 1 or 2 presentations per lecture

4. (Y)our Paper Reports • Start on NEXT Thursday (Sep 6) • Four items: • One good point • One bad point • One potential project (help for proposals) • One question for discussion • Looking for DEPTH • Length is not important; answers that are too long can make it hard to find the depth

5. Discussion Before/After Meeting • Starting the discussion online before the meeting? • Pro: the discussion starts shortly after you read the paper • Con: potential issues with copying ideas or statements for the paper reports • Could use rule that every report needs to address an issue not raised previously • Two NO votes so far

6. Paper Today • Iterative Context Bounding for Systematic Testing of Multithreaded ProgramsMadanMusuvathi & ShazQadeer (PLDI 2007) • Slides from a tutorial by Sebastian Burckhardt, MadanMusuvathi & ShazQadeer • Not only about this specific paper • What would you write for the four items: • One good point • One bad point • One potential project • One question for discussion

7. Paper Overview • Problem • Testing multithreaded code is hard • Trying all thread interleavings doesn’t scale • Solution • Bound the number of preemptive context switches • Tool for Windows: CHESS (stateless, reexecutes) • Evaluation also for ZING (stateful) • Evaluation • 6 programs of various sizes • Found 14 previously unknown bugs

8. Old Questions for Discussion, 1 • Missing by 12:30: [CR], [AS] • Can POR improve CHESS & how much? [JC] • How often CHESS goes to infinite loop in state space graph? [MG] • Does this paper assume correct use of synchronization variables? [AA] • What is the difference between observable and non-observable race? [AA] • Better to limit rather than test threading? [HY] • How to automatically close open systems? [MK]

9. Old Questions for Discussion, 2 • How to know the size of state space? [AL] • Any advantages inserting (removing??) a scheduling point between two syncs? [DM] • Why not bound nonpreempting switches? [JN] • Can iterative context bounding be applied to message passing programs? [ST] • What’s used in industry, stress testing? [QL] • Soundness vs. completeness • False positives vs. false negatives • Precision vs. recall • Combine CHESS and Pex (or X and Y)?

10. Old Questions for Discussion, 3 • Why not report all accessed to shared data not protected by a synchronization variable? • How are data races identified? • How are other bugs identified? Assertions? • What other heuristics can one use? POR? • If you analyze models (ZING), how do you know there is bug in actual code? • How to test a complex multithreaded algorithm in mostly sequential code? Extract it out?

11. Old Questions for Discussion, 4 • How does it reduce time to polynomial? • What about dynamically changing number of threads in programs? Or dynamically created? • Can CHESS handle locks, mutexes, semaphores, etc.? • How to identify preemption or nonpreemption in Java code? • What are limits of this approach? Is it LOC, 10k, 100k, 1M? Is it # of threads, preemptions? • Can it be combined with DFS?

12. Older Questions for Discussion 1 • Which part of JPF/CHESS can be incorrect? • Can a compiler be model checked? • How difficult is it to set environments? • How to “close a system”? • Comparison of testing, model checking, and static analysis • When is model checking applicable? • What kind of bugs it can find? • Why work with language X not Y?

13. Older Questions for Discussion 2 • How well does it scale? • What about memory requirements? • What about time requirements? • How can I apply this on my code? • What’s recent work in this area? • A lot of work in many group, including mine • How to use the tool in practice? • How does CHESS/JPF compare states? • How good are heuristics?

14. Old Sample Project Ideas • Apply CHESS on some software • Did you read the optional reading/report on JPF? • Compare with more tools and techniques • Consider other heuristics for exploration • Message-passing programs? Evolving code? • Automate creation of environments • Extend JPF implementation of CHESS • Contributed by Igor Andjelkovic, Steven Lauterburg, and Mirko Stojmenovic • Reimplement in another model checker • Improve debugging of multithreaded code

15. Next Meeting • Thursday, September 1 • Guest presentation by Vilas Jagannath • Read a paper (listed on Wiki) • Improved Multithreaded Unit TestingV. Jagannath, M. Gligoric, D. Jin, Q. Luo, G. Rosu, and D. Marinov (ESEC/FSE 2011) • NO report due yet • Consider four points: good, bad, potential project, discussion question