Comparison of Student Performance Between Teacher Read and CD-ROM Delivered Modes of Test Administration of English Lang

1. Comparison of Student Performance Between Teacher Read and CD-ROM Delivered Modes of Test Administration of English Language Arts Tests Lisa Harris, University of South Carolina

2. Background • NCLB, IDEIA, Section 504 of Rehabilitation Act • Computer-based accommodations available on national tests • AP exams • SAT • GRE • GMAT, LSAT • Increased computer availability in schools • Increased use of computer accommodations on state-wide accountability tests

3. Past Findings: Where do we go from here? • Read aloud administration • Accommodation or modification? • Other issues • Cuing • Pacing • Computer skills • Reading Level • Item type

4. Past Findings • Read-aloud Accommodation Delivery Options • “Live” reader with written script • Cassette/CD with cassette/CD player • CD played on computer with test booklet • Video taped reader (head shot) • Video taped reader, only text displayed • Video tape with head shot and text • Test on computer with computer generated voice • Test on computer with human voice

5. Call for more research • Studies using math tests • Possibly confounded by lack of content knowledge(Crawford & Tindal, 2004; Bolt & Thurlow, 2006) • Few studies compare delivery methods • Call for more research on read-aloud delivery via computer (Miranda, 2004; Calhoon, 2000) • 1999 Standards for Educational and Psychological Testing • “Support should be provided for any assertion that scores obtained using different items or testing materials, or different testing procedures, are interchangeable for some purposes” AERA, 1999, p.57

6. Research Questions • Does the English Language Arts (ELA) test have the same factorial structure between oral script and CD-ROM modes of test administration? • Controlling for prior ELA performance, are there differences in student performance between test administration modes? • Is there an interaction between test administration mode and student disability? • Does student performance vary between the oral script and CD-ROM modes of test administration? • How do the above results hold across grade levels?

7. Methods: Instruments • Palmetto Achievement Challenge Test (PACT) • Part of the state accountability system • Based on state academic content standards • Grades 3-8 • ELA, math, science and social studies • CDs available in grades 5-8

8. Delivery Methods: CD-ROM • Follow along in test booklet • Record answers in test booklet • No passages or questions appear on screen • Human voice

9. CD Screen Shoots

10. CD Screen Shoots

11. CD Screen Shoots

12. OS Delivery • Teacher-reader follows a script.

13. Methods: Instruments • Depth of Knowledge Rubric for Selected-Response Items • Development • Description • DOK1: Verbatim recall and simple understanding • DOK2:Basic reasoning skills, simple extension beyond what is explicitly stated • DOK3:Complex reasoning • DOK4:Extended reasoning, inference, planning. • Implementation

14. DOK Level Examples • DOK1:Questions focus on verbatim recall and simple understanding. Questions are related to parts of a text rather than the text as a whole. According to the poem, how has the speaker already tried to solve the noise problem? • by telling the mother • by asking Ray to whisper • by giving Ray earplugs • by shutting the bedroom door

15. DOK Level Examples • DOK2: Require basic reasoning skills, comprehension on the literal level, and simple extension beyond what is explicitly stated. Questions require some mental processing of the text or portions of the text. What is the main idea of this passage? A. Louis suffered from poor health. B. Louis could have been a great painter. C. Louis wrote books that many children enjoy. D. Louis used his talents throughout his life.

16. DOK Level Examples • DOK3: Questions require complex reasoning. Questions require an understanding of the text as a whole. When the poet says “Like medals with their ribbons frayed and wavering” (lines 61–62), she is referring to A)  victory B)  fishhooks C)  trophies D)  fish scales

17. DOK Level Examples • DOK4: Questions require extended reasoning, inference or planning. Questions go beyond the literal text and require a deep, purposeful understanding of the text as a whole and/or and understanding of the text as a whole in relation to other texts. Which word best describes the tone of “The Long Hill”? A. fearfulness B. encouragement C. wishfulness D. disappointment

18. Methods: Participants • Students in grades 6-8 who took ELA PACT in spring 2007 using the read-aloud administration

19. Methods: Participants • Due to small sample size for non-LD students disability categories were collapsed into the following: • Category 1: students with learning disabilities • Category 2: all other disabilities including: speech/ language impaired, deaf or hard of hearing, blind and visually impaired, orthopedically impaired, autistic, emotionally disabled, educable mentally disabled, other health impaired, traumatic brain injury, and multiple-disabled due to small sample size

20. Demographic Characteristics of the Students Grade 6

21. Demographic Characteristics of the Students Grade 7

22. Demographic Characteristics of the Students Grade 8

23. Methods: Data Analysis • Measurement Invariance • CFA/ SEM • Step 1: one-factor model was established for each group individually • Step 2: one-factor model was established for each group simultaneously • Step 3: three levels of invariance were tested using 2 : • congeneric (no equality constraints), • tau-equivalent (set factor loadings equal), • parallel (set equal factor loadings and error variances). • DOK1 was selected as the reference variable and the path (1) from DOK1 to the ELA factor was set to 1. • Jöreskog, 1971 • Goodness-of-Fit Indices and Criterion • 2 = chi-squared, insignificant indicates measurement invariance • CFI=comparative fit index ( .95 or above) • SRMR=standardized root mean square residual (.08 or below) • RMSEA= root mean square error of approximation (.06 or below) • Hu & Bentler, 1999

24. 1 x1=DOK1 1 2 x2=DOK2 2  3 3 x3=DOK3 ELA - CD 4 x4=DOK4 4 5 5 x5=CR 6 x6=ER 6 1 x1=DOK1 1 2 x2=DOK2 2  3 3 x3=DOK3 ELA - OS 4 x4=DOK4 4 5 5 x5=CR 6 x6=ER 6 Methods: Measurement Invariance • One-Factor Model for Each Administration Mode • Note.  = independent variable error variance;  = factor loadings; =latent disturbance variance

25. Methods: Data Analysis • Differences in Student Performance • MANCOVA with prior ELA ability as a covariate • Prior ELA ability = total score on 2006 ELA PACT • Dependent variables • DOK1, DOK2, DOK3, DOK4, CR, ER Xijk = μ0 + (Effect of COVAR)+ (Main Effect of MODE) + (Main Effect of TYPE) + (Interaction Effect MODE*TYPE) + Residual • All analyses conducted separately for each grade because the tests are not on the same scale.

26. Fit of One-factor Model Grade 6 Goodness-of-fit Indices of the One Factor Model for Grade 6 Unstandardized (Standardized) Parameter Estimates of the One-Factor Model Grade 6

27. Fit of One-factor Model Grade 7 Goodness-of-fit Indices of the One Factor Model for Grade 7 Unstandardized (Standardized) Parameter Estimates of the One-Factor Model Grade 7

28. Fit of One-factor Model Grade 8 Goodness-of-fit Indices of the One Factor Model for Grade 8 Unstandardized (Standardized) Parameter Estimates of the One-Factor Model Grade 8

29. Summary of Measurement Invariance Findings Note: * p<05

30. Summary of Means by Delivery Mode Grade 6 (question 2a)

31. Summary of Means by Delivery Mode Grade 7 (question 2a)

32. Summary of Means by Delivery Mode Grade 8 (question 2a)

33. Summary of Means by Delivery Mode and Disability Grade 6 (question 2b) • Ranged from 0.01 to 2.9. • These small differences are consistent with the non-significant interaction effect.

34. Summary of Means by Delivery Mode and Disability Grade 7 (question 2b) • Ranged from 0.02 to 0.42. • These small differences are consistent with the non-significant interaction effect.

35. Summary of Means by Delivery Mode and Disability Grade 8 (question 2b) • Ranged from 0.04 to 2.61. • These small differences are consistent with the non-significant interaction effect.

36. Results: Differences in Student Performance

37. Students Scoring At Each Proficiency Level

38. Limitations • Implementation of the CD-ROM format • Opportunity to practice • Broad disability categories • Age and content area • Accommodation decision-making • Other accommodations besides read-aloud?

39. Recommendations for Future Study • Further investigating read-aloud accommodation on specific disabilities • Differences in students who received OS vs. CD-ROM • Looking at DOK differently • Item difficulty • Analyzing complexity of reading passage • Investigating how the CD-ROM was used • Did students repeat questions and passages?