Test-Driven Learning in Early Programming Courses

1. Test-Driven Learning in Early Programming Courses David Janzen California Polytechnic State University San Luis Obispo, CA Hossein Saiedian University of Kansas Lawrence, KS SIGCSE’08

2. I Have a Dream • That all students will some day… • learn to design and test as they learn to program • “... we have as many testers as we have developers. And testers spend all their time testing, and developers spend half their time testing.” Bill Gates in Information Week, May 2002 SIGCSE’08

3. How to Accomplish the Dream? • Test-Driven Development • TDD is a design (and testing) approach involving short, rapid iterations of Unit Test Code Refactor Forces programmer to consider use of a method before implementation of the method SIGCSE’08

4. TDD in Action • class RectangleTest { • @Test • public void testArea() { • Rectangle r = • new Rectangle(5,8); • assertEquals(40, r.getArea()); • } • } • class Rectangle { • private length, width; • public Rectangle(int l,int w) { • length = l; width = w; • } • public int getArea() { • return length * width; • } • } • @Test • public void testPerim(){ • Rectangle r = • new Rectangle(5,8); • assertEquals(26,r.getPerim()); • } • } • public int getPerim() { • return (length * 2) + • (width * 2); • } • } SIGCSE’08

5. Adoption Evolution Predicting the Future TDD Objects Research Labs Dahl & Nygaard NASA Project Mercury Industry Smalltalk, C++ Industry XP Industry Any SW Process “beyond the visionary phase and into the early mainstream” IEEE Software 2007 Grad & Advanced UG Grad & Advanced UG First Year CS First Year CS SIGCSE’08

6. Questions We Want To Answer • Can we teach automated unit testing in first year programming courses without sacrificing other first year topics? • Is it better to teach test-first (TDD) or test-last programming to first year students? • What do students think about TDD? SIGCSE’08

7. How Do We Teach TDD? • Test-Driven Learning involves teaching testing by example XXXXXXXXXXXXX System.out.println(“Result: ” + myObject.calculateResult()); assertEquals(25, myObject.calculateResult()); SIGCSE’08

8. Experiment Design • Proposed TDD/TDL integration into CS1 and CS2 courses at University of Kansas in 2006 as part of PhD research • We don’t always get what we ask for • Given permission to teach two labs and assign two projects • Weeks 6 and 7 in a 16-week CS1 course • Weeks 1 and 3 in a 16-week CS2 course SIGCSE’08

9. Experiment Design CS1 Students Assigned by ID Project 1 solution with tests provided TF/TL Instruction & Survey Project 1 Project 2 Survey 40 40 Test-First Test-Last CS1 66 66 Test-Last Test-First 6 Test-First CS2 30 Test-Last CS2 Students Self-selected Project 1 solution with tests provided SIGCSE’08

10. Experiment Context • CS1 and CS2 both used C++, vi/emacs, and g++ SIGCSE’08

11. Goal: Keep Unit Testing Simple • #include <cassert> • class Exams { . . . }; • int main() • { • run_tests(); • } • void run_tests() • { • { //test 1 Minimum of empty list is 0 • Exams exam1; • assert(exam1.getMin() == 0); • } //test 1 • { //test 2 • Exams exam1; • exam1.addExam(90); • assert(exam1.getMin() == 90); • } //test 2 • } SIGCSE’08

12. CS1 Results: Grades & Effort • Project grades were virtually the same • Time spent on project was ~10% higher for test-first programmers SIGCSE’08

13. CS1 Results: Unit Tests • Test-first programmers wrote more tests on the first project, but fewer on second • Remember that students switched approaches SIGCSE’08

14. CS1 Results: Confidence • No difference in confidence on first project • On second project, test-first programmers were more confident of their solutions (P1 tests) SIGCSE’08

15. CS1 Results: Student Opinions SIGCSE’08

16. CS2 Results: Grades • Project grades were higher for test-first • No significant difference in previous experience or gpa SIGCSE’08

17. CS2 Results: Effort • Time spent on project was lower for test-first programmers SIGCSE’08

18. CS2 Results: Unit Tests • Test-first programmers wrote more tests • Not significant, high standard deviation • Students not graded on tests SIGCSE’08

19. CS2 Results: Confidence • No significant difference in confidence on either project SIGCSE’08

20. CS2 Results: Student Opinions SIGCSE’08

21. 11% vs 63% would choose test-first Reluctance to adopt test-first despite perceived benefits SIGCSE’08

22. Validity Concerns • Study too small to draw any conclusions • Lack of randomization in CS2 • Confounding factors: • New concepts introduced with each project • Solutions with tests provided after 1st project • Simple assert-based unit tests, no framework • C++ • However, many of the results line up with results from industry and more advanced courses SIGCSE’08

23. Conclusions • Unit testing can be integrated into CS1/2 courses without taking time away from other topics • Test-first programmers tend to write more tests • Test-first programmers may do as well (CS1) or better (CS2) on projects • Test-first programmers may spend more time (CS1) or perhaps less time (CS2) on projects • Early programmers are reluctant to use test-first, even after positive experiences with it SIGCSE’08

24. Future Work • Currently conducting a study in CS1.5 • Java and JUnit • TDD from beginning • TDD throughout 10-week quarter • Early Results • TDD takes no extra instruction time • Existing course materials can be rewritten w/TDD • TDD could influence course refactoring • e.g. delay coverage of I/O SIGCSE’08

25. Questions • Resources at http://www.simexusa.com/tdl/ • If you are an early adopter of TDD in CS1/2, I’d like to know, email me at djanzen (at) calpoly.edu • Acknowledgements • SIGCSE Special Projects Grant – seed funding • Lockheed Martin – current (future work) study SIGCSE’08