Introductory Programming, Criterion-Referencing, and Bloom

1. Introductory Programming,Criterion-Referencing, and Bloom Raymond Lister & John Leaney University of Technology, Sydney Australia

2. McCracken, et al. • Insert their graph here • points to make ... • Multi-institutional! • weaker students flounder • ... Leading to quote on next page ...

3. Buck & Stucki, SIGCSE 2001 Students who are required to write complete programs as lab assignments are ... “... overwhelmed, uncertain of how to begin, and grasping at the air … [leading] … to the self-destructive tendency to do experimental programming, where they just randomly throw things in to see if it helps”.

4. McCracken, et al. After their first “clean compile”, the weak students ... “... are then surprised by what the program really does when presented with data”.

5. Norm-referenced Assessment • Same assessment activity for all students • Students graded “to a curve” • Aimed at the “middle student”? • Weak students flounder ... • ... but pass anyway? • Strong students do well ... • ... but were they challenged?

6. Criterion-Referenced Assessment • Qualitatively different assessment activities for each grade. • “Truth in sentencing” • Weak students learn something, without holding back strong students. • Strong students are challenged without hurting the weak students.

7. Grades & Bloom’s Taxonomy “A” Bloom’s Synthesis & Evaluation level “B” “C” Bloom’s Knowledge & Comprehension level The message of Bloom: test at all levels of the taxonomy. Bloom’s Application & Analysis level

8. “C” and Bloom’sKnowledge/Comprehension (1) • No requirement that a ‘C’ student manifest any sort of design skill. • Lab Exam: Translate pseudo-code of a complete Java class into correct, readable code that compiles. • For example ...

9. For example ... • Declare a header for a public class, called "BMIApplet", a subclass of Applet, which implements the methods prescribed by the "ActionListener" interface. • Declare a header for a public constructor, which takes zero parameters. • Declare an integer "BMI” and assign a value to "BMI", by calling "computeBMI", a method in this class.

10. “C” and Bloom’sKnowledge/Comprehension (2) • The ability to read code, and demonstrate an understanding of what that code does. • Tested by multiple choice questions. • Pass mark is a “mastery” 70%, not 50%. • For example ...

11. For example ... The code for a class called “Counter” is given as follows (consider carefully the use of the “static” keyword). public class Counter { private static int count = 0; public void increment( ) { ++count; } public int getCount( ) { return count; } } If the following code is executed: Counter counter1 = new Counter( ); Counter counter2 = new Counter( ); counter1.increment( ); counter2.increment( ); int total = counter1.getCount( ) + counter2.getCount( ); the value in “total” is: a) 0 b) 1 c) 2 d) 3 e) 4

12. Rote Learning for a “C” ? • Approximately 30 “bottom passing” students with mark in the multiple choice exam of 70-77% • On questions with previously seen code, 63-89% of bottom passing students got the question right, median 81%. • On questions with unseen code, 63-85% of bottom passing students got the question right, median 78%. • Conclusion: Not rote learners!!!

13. 30% fail the multiple choice exam at their first attempt. Truth in sentencing: If some students can’t even read code, how can we expect them to write code? Why assess them by asking them to write code? The message of Bloom: test at all levels of the taxonomy.

14. The Story So Far “A” Bloom’s Synthesis & Evaluation level “B” “C” Completion of pseudo code; multiple choice questions Bloom’s Application & Analysis level

15. “B” and Bloom’sApplication/Analysis (1) Bloom distinguishes between comprehension and application as follows: “A demonstration of comprehension shows that the student can use the abstraction when its use is specified. A demonstration of application shows that he (sic) will use it correctly, given an appropriate situation in which no mode of solution is specified.”

16. “B” and Bloom’s Application/Analysis (2) • No surprises here ... • Assessed traditionally, with an assignment. • “Pig World” • Third of class opt out and happy with “C”.

17. The Story So Far “A” Bloom’s Synthesis & Evaluation level “B” “C” Completion of pseudo code; multiple choice questions Traditional assignment

18. The “A” Part 1: Bloom’s Synthesis level Bloom defines “Synthesis” as ... "Skill in writing, using an excellent organization of ideas and statements" (p.169). (essays? programs?) “... ability to write creatively a story, essay, or verse for pleasure, or for the entertainment or information of others" (p. 169).” assessed by a project ...

19. The “A” Part 1: Bloom’s Synthesis level (cont.) “Write a program. You decide what it does.” To avoid a token effort, we specify: - at least 100 lines of code. - at least 4 classes - etc. We want to let HD students to learn-from-play. Constructivism rules!

20. The “A” Part 2: Bloom’s Evaluation level Bloom defines “Evaluation” as ... "the making of judgments about the value, for some purpose, of ideas, works, solutions … ” (p.185). Assessed by peer review. - I provide some vague evaluation criteria. * clarity of code. * quality of user interface.

21. Conclusion “A” Self-selected project; peer review. “B” “C” Completion of pseudo code; multiple choice questions Truth in sentencing The message of Bloom: test at all levels of the taxonomy. Traditional assignment

22. Survey Results (1) "My learning experience in this subject were interesting and thought provoking". Agreed 69% Neutral 18% Disagreed 13%

23. Survey Results (2) "I found the assessment fair and reasonable". Agreed 64% Neutral 30% Disagreed 7%

24. Grade Distribution Grade After After First MCQ Exam Second MCQ Exam Fail 29% 10% C 18% 37% B 47% as before A 6% as before