Use SP Xpress to analyze students’ performance & evaluate the question items setting

1. Use SP Xpress to analyze students’ performance & evaluate the question items setting St. Anthony’s School 16 Jan 2013

2. Background information Curriculum Assessment

3. Enhance the effectiveness of learning and teaching What to start How to start

4. Evaluation after Assessment Why did students choose A, not B, C or D? Are the questions too difficult or too easy? What are the follow-up actions? Have the students learned? Have they studied hard?

6. Data – Good to start • Student MCI • Item MCI • Cronbach Alpha • Graph shape

7. 2 Modified Caution Indexes Students’ Performance Student Modified Caution Index 0> MCI> 0.3 Question items’ Performance Item Modified Caution Index 0> MCI> 0.3

8. 2012-13 P.4 1st Reading Exam Student MCIxP Chart

9. 2012-13 P.4 1st Reading Exam Student MCIxP Chart

10. More focused strategies

11. Item MCI xP Chart 2012-13 P.4 1st Reading Exam

12. Item MCI xP Chart Need improvement Difficult questions Problem questions

13. MCI item P-value chart According the Item Modified Caution Index, most of the items set are normal.

14. Cronbach Alpha (Reliability)

15. 2011-2012 P.3 First Exam Slightly Easy

16. 2012-2013 P.4 First Exam Appropriate

17. Which question items should be improved?

18. Which question items are difficult to our students?

19. SP Xpress analysis on ‘2011-2012 Primary 3 Third Exam Paper’ • SP Xpress analysis results: • Type A: 20 items • Type B: 1 item • Type C: 1 item • Type D: 3 items • Let’s examine one item from each category (Type B, C, D)

20. Exam item example: Type B On which floor of the centre, can Grandpa find presents for Jack?A. 1/F B. 2/F C. 3/F D.4/F

21. Possible reasons for item being Type B • Type B: Item is not difficult but requires amendments because many students couldn’t answer it. • Possible reasons: • Item design flaws • Not close to what teacher taught in class • Students lack specific training in reading comic strips

22. Teacher examined the Type B item • Students found it difficult because … • Need to process and connect many ideas all at the same time

23. Recommended solutions • Increase students’ chances in reading English comic strips in classroom teaching or leisure reading • Teacher needs to illustrate more clearly in connecting ideas • Teacher needs to teach more explicitly which means more visual demonstrations

24. Exam item example: Type C In which step does Jack’s mum add ‘soy sauce’ and ‘wasabi’ to the sushi? A. Step 1 B. Step 3 C. Step 6 D. Step 10

25. Possible reasons for item being Type C • Type C: Item is difficult and requires amendments because many students couldn’t answer it. • Possible reasons: • Confusing description • Question misleading students • Question containing error(s)

26. Teacher examined the Type C item • Students did not have the relevant personal experience in making and/or eating sushi • The recipe did not mention dipping sushi into ‘soy sauce’ or ‘wasabi’ after the finish • Without experience, students could not interpret the correct answer which is the final step • This question contains the error which requires outside knowledge uncommon among the students

27. This is the answer but the option rephrased the instructions. Exam item example: Type D After cutting the vegetables and crab meat, Jack’s mum ____.C.takes some rice wine vinegar and mixes it with the sushi rice

28. Features of Type D item • Type D: Item is difficult and doesn’t require amendments. It is suitable for screening the high ability students. Low and mid- ability students can’t answer it. • Type D items should be kept at a low proportion in an assessment. • The rephrasing is difficult for students to interpret the correct answer.

29. Analysis of TSA P.3 Reading Paper • Predict P.3 students’ results in the upcoming TSA reading assessment • Compare the overall students’ ability from different years • Examine the neglected elements in our teaching • Assumption: TSA assessment covers the basic competency that all students are supposed to have acquired by the end of a learning stage

30. SP Xpress analysis on ‘2011 TSA P.3 Reading Paper (3ER1)’

31. An Interesting Fact: A common Type B item across 2 years • Part 1B Question #1How many chapters are there in this book? • ‘Chapter / Table of Contents’ can be an obstacle • Students never encounter ‘Chapter’ in storybook teaching (replacing textbook teaching)

32. An Interesting Fact: A common Type B item across 2 years • Sam buys two sandwiches with the Rainbow Fast Food coupon. How many sandwiches can he get?C. three sandwiches (= Buy 2 + 1 Free) • Students are not familiar with ‘Buy 2 Get 1 Free’ concept even though they can understand every single word

33. An Interesting Fact: A common Type D item across 2 years • “… too. In the end, John wins the race. All the people cheer loudly …” • Read lines 11-12. What do the people do when John wins the race?C. shout happily • Students are not familiar with synonyms cheer loudly shout happily

34. Personal reflection • Teachers become aware of the quality of the assessments • Accumulate professional experience in setting better assessments • Teachers prepared for the unexpected and unlearned • Teachers become cautious and adjust their teaching

35. Things to do • Suggest Ts reflect on their exam paper using SP Xpress • Facilitate Ts process the data with technical supports • Set up a school based question bank • Nurture a culture on giving a more constructive feedback to both Ts and Ss after assessment

36. Thank you so much for your time!