Analysis of BABELnot MCQs for Level 1, 2 Exams

Analysis of BABELnot MCQs for Level 1, 2 Exams Daryl D’Souza, James Harland, Margaret Hamilton BABELnot Melbourne, 20 July 2012

Research Background • Research aim: AnalyseBABELnot MCQs in written exams at Levels 1 and 3. These questions employed in thinking-out-aloud experiments (ref: ??) • Research questions: How do the students’ performance on BABELnot questions correlate with the overall performance on the exams?

Methodology The exams • Programming 1 (Level 1) • Semester 1, 1st year, CS/SE • No previous programming (strictly) necessary • Java language • 20 MCQs, 5 SAQs, 1 3-part themed question • All three BABELnot Qs used as MCQs • Programming Techniques (Level 3) • Typically 2nd year • Prerequisite Programming 2 • C language • Two BABELnot Qs used • one as one part of a seven-part SAQ • one SAQ on its own

Question 5 P1 boolean testArray(int [] x, int arrayLength) { for (int i = 0; i < arrayLength-1; i++) { if (x[i] > x[i+1]) return false; } return true; } If testArrayreturns true after this code is executed, which of the following is the strongest statement we can make about the contents of the array x? Assume the subscripts p, p+1 and q are legal indexes of x. *A. x[p] <= x[q] for all p < q. B. x[p] <= x[p+1] C. x[p] <= x[p+1] where p is an even number (i.e. 0, 2, 4, etc). D. No statement can be made, as the last iteration of the loop will attempt to index an element of the array that does not exist. E. x[0] < x[arrayLength-1]

Question 11 P1 How many times will the while loop body below be executed? The code below in the right of the table moves all elements of the array x one place to the right, with the rightmost element being moved to the leftmost position. The variable length contains the number of elements in the array x: Consider the partial code provided in the above table, on the left. When the occurrences of ??? are replaced with appropriate code, that code can undo the effect of the code on the right. That is, when the ??? are replaced appropriately, the code can move all elements of the array x one place to the left, with the leftmost element being moved to the rightmost position. The table already shows that line 3 is different in the code on left to the code on the right. When the ??? on lines 1, 2 and 4 are replaced with appropriate code, which of these lines of code must be different between the left and the right A. None of the lines 1, 2 and 4 must be different. B. Only lines 1 and 4 must be different. C. Only line 2 must be different. *D. All of the lines 1, 2 and 4 must be different.

Question 12 P1 Below is incomplete code for a function which returns the minimum value in the array x. When an appropriate line of code is selected from each box, the completed code will scan across the array, using the variable minsofar to remember the best candidate for minimum so far. int min(int x[], int arrayLength) { int minsofar = ; for (int i = 1 ; i < arrayLength; i++) { if ( x[i] < ) { minsofar = ; } } return ; } Which of the following choices of lines from the boxes will produce a correct version of the function min? A) Lines ACFG only. B) Lines ADEH only. C) Lines BCFG only. *D) Both lines ADEH and lines BCFG. 0 x[0] minsofar x[minsofar] i x[i] minsofar x[minsofar]

Question 1g PT In one sentence, explain in plain English the purpose of the following piece of code. int fn(int *array, size_t n) { size_ti; for (i = 0; i < n - 1; i++) { if (array[i] > array[i + 1]) return -1; } return 1; }

Question 4 PT The purpose of the block of code below is to take an array of integers and move all elements of the array one place to the right, with the rightmost element moving around to the leftmost position. void shift_right(int *array, int n) { int temp = array[n - 1]; int i; for (i = n - 2; i >= 0; i--) array[i+ 1] = array[i]; array[0] = temp; } Write a function shift_left that will move all elements of the array one place to the left, with the leftmost element moving around to the rightmost position. void shift_left(int *array, int n) { /* . . . */ }

Results—P1

Results—PT Chi-squared: p = 0.20 Chi-squared: p = 0.228

Discussion • The only significant correlation was between the BABELnot questions and BABEL ones on the PT exam • No significant correlations on P1, possibly due to being only 3 MCQs and less overall marks than on PT. • BABELnot questions tended to ‘spread’ the students out more than other questions (high mode & median, lower mean)

BABELnot Deliverables • If Mercy's work contributes to BABELnot, then it must be contributing (perhaps indirectly) to one of the projects' deliverables ...• A formalism for describing learning outcomes, exam assessment, and the method for mapping between the two, for the first three programming subjects• The learning outcomes of the first three programming subjects, from at least the six participating universities, re-expressed in the formalism• An archive of exam questions, with meta-tags mapping the questions to the formalism, serving as examples to other academics• Performance data from real students for a subset of the archived exam questions

Mercy’s work • Briefly Mercy’s work involves the creation of Facebook sites with the BABELnot questions posted there. We aim to find out: • Why students make the errors they do • What mental models students have about these questions • What words they are using to explain their answers and/or difficulties • Find out in their own words how most students perform while answering these exam questions

Contribution to BABELnot • Performance data from real students for a subset of the archived exam questions • These are real students enrolled in the programming courses • They must solve the actual subset of BABELnot questions • They must explain their steps somehow – to the group, and not to the instructor alone, as in a pencast. • A formalism for describing learning outcomes, exam assessment as in their discussions, students will need to identify where they are struggling, and this discussion can: • improve our formalism and language • identify the problematic learning outcomes eg: can we simply say one outcome would be to compute a ‘sort’ if we are discussing objects?

Analysis of BABELnot MCQs for Level 1, 2 Exams