CCSSO, New Orleans June 25, 2014

1. Assessing PARCC Mid-flight A Qualitative Analysis of Item Types on Tablets and ComputersEllen Strain-Seymour and Laurie Davis (Pearson) CCSSO, New Orleans June 25, 2014

2. Tablet/Computer Comparability A two-part study to understand device comparability: students taking the PARCC test online should receive no advantage or disadvantage based on their device • Part I (July 2013): hybrid of usability study and cognitive lab focusing on item types • What factors threaten device comparability, and how can we resolve these issues before the field test? • Part II (concurrent with spring 2014 field test): quantitative comparability study • Have we resolved these issues?

3. Research Questions • Can students use the interfaces for technology-enhanced items (TEIs) to create responses that reflect their knowledge and skills? • How does language usage, formatting, or other aspects of content design impact student performance on TEIs? • For these item types, do usability differences exist between tablets and computers? • For more complex item interactions, does prior exposure to a tutorial on item type functionality improve students’ ability to use an item type interface to create responses without difficulty? Other Areas for Investigation: • Differences by age, tablet familiarity, and content knowledge • Student use of scratch paper and styluses

4. Participants 72 Students, 2 Locations, 3 Grade Levels, 4 Conditions Tutorial A covered half of the more complex item types. Tutorial B covered the other half.

5. Tutorial Example: Polygon Graphing

6. Tutorial Example: Composite Graphing

7. Tutorial Example: Fraction Modeling

8. Tutorial Example: Function Graphing

9. Tutorial Example: Text Extractor

10. Included Item Types

11. TEI Types: Interactive Number-line

12. TEI Type: Zoom Number-line

13. TEI Type: Shape Transformation

14. Findings: Validity & Usability Most severe examples that could have a performance impact: • Draggers were large and drop-zones were small. Nearly all students struggled to drag objects to drop-zones. These items could be easily redesigned to avoid this problem. • Suppressed lines beneath labels on coordinate grids led a few students to count cells incorrectly. • Lack of marking tools led to students doing in situ operations “in their head” or on scratch paper. Led to incorrect answers in some cases. One scratch paper user transcribed information incorrectly. • Can students use the interfaces for technology-enhanced items (TEIs) to create responses that reflect their knowledge and skills? • Yes, in general. • Some usability problemsdoexist.

15. Findings: Predicting Student Struggles • When there was not a significant usability problem (e.g., the drag-and-drop item with small drop-zones and large draggers), the two biggest determining factors for predicting when students would experience difficulty using the interface to construct a response: • When students received a tutorial on a given item type, they were generally able to focus primarily on content issues and use the interface appropriately. • When students had the appropriate content knowledge, they were generally able to use the interface to generate a response. In many cases, student comments suggest that they visualized the correct response after reading the prompt and were motivated to use the interface to construct that response. For students who did not appear to have the content knowledge, it was hard to differentiate their struggles with the content from their struggles with the interface. Some student responses suggest that they were looking for the interface to help guide them in the way that multiple-choice responses can.

16. Findings: Tutorials • For more complex item interactions, does prior exposure to a tutorial on item type functionality improve students’ ability to use an item type interface to create responses without difficulty? • Yes, tutorials (practice items that provide instruction as to use) appear to be helpful. The format used helped students to experiment with functionality and guided them to certain useful discoveries. • Some aspects of the tutorial format could be improved. • Language could be simplified. • Information could be presented through multiple modalities. • Steps should be broken down into smaller pieces, so that there is less text to consume on a single screen. • Students should have a mechanism for evaluating their understanding of functionality.

17. Findings: Usability Comparisons For these item types, do usability differences exist between tablets and computers? • A small number of minor usability issues seemed to be isolated to the tablet: inadvertent triggering of zoom or the magnified text selector; temporary uncertainty caused by device latency (decreased responsiveness). • Most usability issues were similar to those experienced on the computer, but… • These usability issues occurred with greater frequency on the tablet (perhaps due to less user feedback via cursor icons and hover effects). • Usability issues were more difficult to recover from on the tablet. Students tended to tap more aggressively or double-tap, which then led to inadvertent zoom, the appearance of the text selector, or immediate undoing of the action just performed.

18. Overall Recommendation Since a lack of familiarity with item types can lead to difficulties in using the interface to construct a response (particularly for tablet users who do not receive user feedback via roll-overs and cursor icons) and since students taking the test on tablets may be more affected by minor usability problems, the best route to device comparability is to strengthen plans for tutorials and resolve usability issues.

19. Other Findings • Younger students tended to struggle more to learn new interfaces and had more difficulty with language used to describe interface features. • Students easily interpreted “click” to mean “tap” when on a tablet, such that converting all language in items may not be necessary. • Specific language (e.g., “click to highlight”) within item instructions was more useful than generic language (e.g., “select,” “highlight”). Bolded references to the number of selections required (e.g., “choose two”) were helpful to students. • A small number of students find styluses to be very useful, even when increased precision from the stylus was not immediately apparent. • Scratch paper is essential for some items. • Some items used in the study would have been more usable if delivered using a different item type. Usability guidance on such issues should be documented and provided to content developers.

20. Limitations • The effect of tablet familiarity was hard to evaluate, since the participants did not use tablets in the classroom. This failure in recruitment was due to difficulties accessing students during the summer. • Qualitative research can help identify usability issues and other potential threats to validity, but quantitative research is necessary to understand whether such issues translate to statistically significant impact on student performance. • Student participants had not transitioned to Common Core State Standards and may have had gaps in content knowledge as a result. • Students only encountered one or two items of a given item type. A learning effect may increase the ease of their engagement with TEI interfaces once a larger number of items have been experienced.

21. Sample Recommendations • Sample test delivery system recommendations: • Redesign calculator icon for greater recognizability • Make the scrollbar wider, more similar to standard computer scrollbars • Change labels on passage paging/scrolling interface to "page up" & "page down” • Include a counter that tracks the number of highlights applied in hot text items • Sample item development, lay-out and process recommendations: • Use specific terminology such as "click to select," "click to highlight," "drag” • Inform students in item instructions when all draggers need not be used • Consider integrating a usability review as part of item development • Use bolded words when a certain number of selections or highlights are required • Avoid having drag-and-drop components not fit within a single screen • Sample policy/preparation recommendations: • Allow scratch paper and stylus use • Make tutorials available

22. Why and What’s Next? • Reasons for including qualitative studies (cog labs, usability studies) before quantitative comparability studies • Ability to locate issues and make changes before comparability studies for the best chance at comparability • Far less expensive…usability engineering rule of thumb: with 8-10 participants 90% of usability issues discovered • Even in cases where fast corrections are difficult, the reason behind mode effects or unexpectedly low student performance may have been discovered in cog lab • Usability issues affecting all devices can be discovered

23. Questions? For more information: ellen.strain-seymour@pearson.com http://researchnetwork.pearson.com/resources/tablets