Testing Information Literacy in Digital Environments: ETS’s iSkills Assessment

1. Testing Information Literacy in Digital Environments:ETS’s iSkills Assessment Irvin R. Katz Research and Development Division Educational Testing Service http://www.ets.org/iskills

2. Technical Literacy Information Literacy ICT Literacy: a bridge between Information and Communication Technology Literacy • Can I find information in a library database? • Can I create a persuasive presentation? • Can I identify conflicts of interest in a web site? Database Word Processing Presentation Access Evaluate Use • Can I bold a word? • Can I open a database? • Can I find information? • Can I evaluate authority? (c) 2007 Educational Tesing Service. All Rights Reserved.

3. ICT Literacy Model Define Access ICT Literacy:Using cognitive skills in a technological environment Evaluate Manage Integrate Create Communicate (c) 2007 Educational Tesing Service. All Rights Reserved.

4. ICT Literacy includes: • Define: Formulate a research statement to facilitate the search for information • Access: Find and retrieve information from a variety of sources • Evaluate: Judge the usefulness and sufficiency of information for a specific purpose • Manage: Organize information so as to find it later • Integrate: Summarize or otherwise synthesize information from a variety of sources • Create: Generate or adapt information to meet a need, expressing a main point and supporting information • Communicate: Adapt information for a particular audience (c) 2007 Educational Tesing Service. All Rights Reserved.

5. Design an assessment with face validity Authentic, relevant, performance-based Measure the application of knowledge, rather than facts or standard procedures Aligned to nationally-recognized standards Ensure that it is based on solid measurement principles Quality & Fairness Validity Reliability Provide students, instructors, and institutions with useful data and feedback Challenges in developing a measure of ICT Literacy (c) 2007 Educational Tesing Service. All Rights Reserved.

6. Assessment Development Process and Timeline • 2001-2002 – Convened International Panel • Published Digital Transformation • 2003–Evidence-Centered Design of iSkills framework • Identified Test Purpose & Uses • Designed Proficiency Model • Created Evidence Models • Developed Task Models & Defined Scoring Rules • 2003-2004: Develop, conduct field trials, and pilot iSkills • 2005 – Delivered Institution-Level Test • 2006 – Delivered Individual Tests (Core & Advanced) (c) 2007 Educational Tesing Service. All Rights Reserved.

7. ETS iSkills Assessment • Interactive, web-based, and performance-based, presenting real-world scenarios – NOT multiple-choice • Designed to measure higher-order problem-solving and critical thinking skills • Includes 14 short (3-5 minute) tasks and 1 longer (15 minute) task • Delivered in two sessions totaling 75 minutes (c) 2007 Educational Tesing Service. All Rights Reserved.

8. Two levels for two transitions • Advanced Level • Do students have the requisite ICT Literacy skills by their 2nd year of postsecondary education? • Targets students transitioning to upper-division coursework or entry-level workforce • Core Level • Do students have the requisite ICT Literacy skills by their last year of secondary education? • Targets students transitioning to first year of postsecondary education • Measure the same skills • 75 minutes of testing time (divided into two sections) (c) 2007 Educational Tesing Service. All Rights Reserved.

9. iSkills Example Tasks and Scoring

10. Automated Scoring Algorithms • Within each task, scores are assigned for each identified response • Each observation tied to one of the seven performance areas • Most observations scored as correct (1), partially correct (0.5), or incorrect (0) (c) 2007 Educational Tesing Service. All Rights Reserved.

11. Developing the Scoring Rules • Experts specified general characteristics that distinguish levels of ability • Scoring of a particular response combines • Expert judgment • Demands of task • Availability of data (responses and logs) (c) 2007 Educational Tesing Service. All Rights Reserved.

12. Example: Extended Search & Evaluation • Long task: Fifteen minute duration • Assesses two ICT Literacy skill areas • Access: The ability to collect and/or retrieve information in digital environments. • Evaluate: The ability to determine the degree to which digital information satisfies the needs of the task in ICT environments. (c) 2007 Educational Tesing Service. All Rights Reserved.

13. Examinees conduct a simulated web search to locate a website that illustrates the information need: a recent example of excessive journalism (c) 2007 Educational Tesing Service. All Rights Reserved.

14. The search query entered – the terms and Boolean operators – provides evidence of how well students pull out the key elements of the information needed. (c) 2007 Educational Tesing Service. All Rights Reserved.

15. The search results that students investigate, and the ones they select as research sources, provide evidence of their understanding of authority, objectivity, and timeliness of on-line sources of information. (c) 2007 Educational Tesing Service. All Rights Reserved.

16. Specificity of Search Terms (c) 2007 Educational Tesing Service. All Rights Reserved.

17. Quality of Search Results Web search universe is a tailored collection of pages designed to vary in relevance as well as other criteria (c) 2007 Educational Tesing Service. All Rights Reserved.

18. EvaluateScores Authority, relevance, objectivity, and currency of web pages selected, scored on: • Were the correct pages selected? • Were primarily relevant and trustworthy pages visited from the search results? (c) 2007 Educational Tesing Service. All Rights Reserved.

19. Thank you For more information: http://www.ets.org/iskills For an on-line demo: http://www.ets.org/Media/Products/ICT_Literacy/demo2 (c) 2007 Educational Tesing Service. All Rights Reserved.