Progress Monitoring of Oral Language Using the Narrative Language Measures

1. Progress Monitoring of Oral Language Using the Narrative Language Measures Douglas B. Petersen, PhD, CCC-SLP University of Wyoming

2. Objectives • Describe the key features of a curriculum-based measure • Explain why narrative assessment is suitable for progress monitoring • Describe the purposes of the NLM • Administer and score different subtests of the NLM • Describe how the NLM can be integrated into a multi-tiered system of support for language

3. Defining the Problem • "How we define a problem usually determines how we analyze it. It sends us in a particular direction. • And how we analyze a problem—the direction we take —absolutely determines whether we find a solution and what the quality of that solution is.” -Morgan D. Jones

4. National Assessment of Educational Progress (NAEP, 2011)

5. Broad View of Reading Perfetti, 1985

6. Word recognition Linguistic Comprehension Narrow View of reading Reading Language Kamhi, 2009

7. Common Core State Standards • “Children’s oral language competence is strongly predictive of their facility in learning to read and write… • listening and speaking vocabulary and even mastery of syntax set boundaries as to what children can read and understand no matter how well they can decode.” (Appendix A, pg. 26)

8. Problem

9. Direction - Solution • To identify students who experience language difficulty so we can intervene and promote academic success as early as possible • Language Difficulty • Not just a language disability • Includes English language learners • Includes cultural diversity

10. Multi-tiered System of Support Assessment Assessment Assessment Assessment

11. A Paradigm Shift • Purpose has shifted from eligibility and classification to early detection, prevention and progress monitoring. Not QUALIFY-PLACE-SERVE But TEST-TEACH-TEST

12. Purposes of Assessment • Screening (Universal) • Helps answer “Who is at risk?” • Diagnosis and Eligibility • Helps answer “Does this student have a disability?” • Instructional/Intervention Planning • Helps answer “What and how should we teach?” • Progress Monitoring • Helps answer “Is our intervention working?” and “What, if anything, needs to be adjusted?”

13. Curriculum Based Measurement • Specific set of procedures for measuring student academic growth • Designed to evaluate the effectiveness of instruction • Primarily used for • screening • progress monitoring • Often interpreted to mean the same thing as General Outcome Measurement (Deno, 2003)

14. CBM Drives RTI • Screening (Benchmark Assessment) • Universal benchmark screening (e.g., Fall, Winter, Spring) • Students’ scores are compared to a performance criterion • Identifies children who may require intensified services or more frequent and targeted monitoring • Progress monitoring • Regular sampling of performance (e.g., weekly, monthly) • Data are used to make decisions about • Adjustments in intervention • Entry into and exit from tiers • Need for intensive “special” education

15. Requirements for Curriculum Based Measurement • Strong reliability and validity • Time efficient/easy to administer/score • Alternate forms of standardized tasks • Measure socially important outcomes • Sensitive to growth due to intervention -Deno, 2003

16. Requirements for Language RTI • An effective, manualized language curriculum that is easy to use and sufficiently flexible to address the language needs of diverse children • Valid, reliable, easy to use, assessment tools with equivalent forms for frequent sampling of students’ language skills and measuring growth over time. +

17. Claiming the Job • “There is an urgent need for a simple, quick, standardized measure of language that can be administered repeatedly…in a manner akin to the continuous ‘dibbeling’ that is used to monitor progress in code learning.” • “It behooves the profession and research community to claim the job of developing a set of very brief, standardized benchmark measures of vocabulary, grammar, and narrative…” Ukrainetz, 2006

18. Narrative Language • Narrative – an account of causally-related events in temporal order (aka. a story) • Narrative language is associated with academic achievement. • Narrative proficiency at age 5 is one of the best predictors of future reading comprehension(Catts, Fey, Tomblin, & Zhang, 2002; Dickinson & McCabe, 2001; Griffin, Hemphill, Camp, & Wolf, 2004.)

19. Narrative Language • Narratives are socially important. • Both language comprehension and production can be measured through narration. • Narration is a authentic, functional integration of language • Narratives …can be the intervention target …can be the intervention context

20. Reading Standards for Literature

21. Analysis of Narratives • Story Grammar Analysis • Entire story is the unit of analysis • Story Grammar • Linguistic Complexity Analysis • Words and clauses are the units of analysis • Linguistic Complexity

22. Story Grammar • Story Grammar: Main parts of the story • causally and temporally related • Reveal the plot or point of the story • Reinforced and tested in American schools

23. Story Grammar Character Problem Feeling Action Ending Setting End Feeling

24. Linguistic Complexity • The storyteller must create a context for the listener almost exclusively through the use of sophisticated, complex language.

25. Linguistic Complexity • Similar to written language • Adverbs (e.g., quickly) • Temporal subordination (e.g., When he was walking…) • Elaborated noun phrases (e.g., The huge dog that was scary) • Mental & linguistic verbs (e.g., decided, considered) • Conjunctions (e.g., and, but, because) • Dialogue (e.g., He said, “I need a band-aide!”)

26. Quality Narratives • A Quality Narrative… • includes important Story Grammar elements that create an episode (plot) • Initiating Event (problem) • Attempt • Consequence • includes complex language features that convey causal and temporal relations between key events (Ukrainetz, 2006). Quality is impacted by components and the clarity of the relations between components

27. Narrative Language Measures (NLM) • Three Subtests • Test of Narrative Retell (TNR) • Test of Personal Generation (TPG) • Test of Story Comprehension (TSC; Preschool only) • Preschool, Kindergarten, First, Second, Third • 25 equivalent stories per grade level • 9 Benchmark stories (3 Fall, 3 Winter, 3 Spring) • 16 Progress Monitoring stories

28. Equivalent Features of the NLM • Consistent across: • Story Grammar • Linguistic Complexity (in extreme detail) • Consistent vocabulary complexity with limited overlapping vocabulary (across stories and across grade levels)

29. Technical Adequacy • Because the TNR is a new measure, empirical investigations of its technical adequacy are ongoing. • The following table outlines the available psychometric characteristics.

30. Standardized Features of the NLM • Standardized administration procedures • TNR • TPG • TSC (Preschool only) • Standardized scoring guidelines • Real time • Audio file • Transcript

31. Standardized Administration • Select the appropriate level • Benchmark TNR Purpose: compare children to an age or grade criterion • Progress Monitoring TNR Purpose: show an individual child’s language growth over time Materials and Setting • The booklets are designed to be administered in order • The progress monitoring stories can be administered in a different order or a story can be skipped. • Stories should never be repeated

32. Standardized Administration Examiner says, “I’m going to tell you a story. Please listen carefully. When I’m done you are going to tell me the same story. Are you ready?” Examiner reads the selected story word for word in a moderate pace with normal inflection. Examiner says, “Thanks for listening. Now you tell me that story.” Examiner should wait 10-15 seconds before providing encouragement. If the child is reluctant, examiner encourages by saying “It’s okay. Just do your best.” If the child asks for help, examiner says, “I can’t help but you can just tell the parts you remember.” Examiner listens and makes only neutral comments such as “uh-huh”and“okay.” Examiner may not repeat the story or any part of it, but can repeat the directions and provide encouragement as needed. When the child appears to be finished, the examiner says, “Are you finished?” If the child is not finished, the examiner should let the child continue. If the child does not tell a story even after encouragement the examiner says, “That’s okay. Thanks for listening to my story.”

33. TNR:P with Pictures • Lay out pictures before beginning the story. • As you read the story, point to the picture that depicts the part of the story.

34. Time Efficient, Economical Brief Administration • Three benchmark TNRs take about 5 minutes • A single TNR for progress monitoring takes less than 2 minutes. Reduced Scoring Time • Scoring can be done in real-time while the child is retelling the story. • Scoring can be done by listening to an audio recording

35. TNR Scoring in Real-Time

36. TNR Scoring in Real-Time • Um, This kid, um Shane was getting ready for school because he was late and he couldn’t find his shirt so then he asked his mom to help him find it and she told him it was on the dryer and he got it and he was happy.

37. TNR Scoring in Real-Time 2 4 18 12

38. Some TNR Scoring Guidelines • While listening to a child retell a story, the examiner cues in on key words or phrases provided in the scoring form. • To be scored, the child’s utterances do not need to match the samples in the scoring rubric exactly or word for word; however they should be very close and contain the same amount of correct information. • To earn 2 points, the utterance should be able to stand‐alone. The scorer should not need to have shared knowledge of the story to correctly interpret the utterance. • When children say something other than the exact words and phrases contained in the scoring form, examiners should write in what they say. • It is helpful to underline or write words and phrases as children say them and once the administration is complete, rate the words or phrases as 2, 1, or 0 based on the general scoring guidelines. Pgs. 7 & 9

39. Test of Personal Generation (TPG) • Assesses language using a conversation elicitation technique • (adapted from McCabe & Rollins, 1994). • Examiner elicits personal stories while playing with a young child in a very natural manner. • Provide at least 3 opportunities. • Most difficult narrative task we use.

40. TPG Variation • Immediately following a child’s retell narrative (TNR), elicit a personal generation by asking, “Has something like that ever happened to you?” • Samples a different dimension of their narrative skills in a very efficient manner • Content limitations • Motivation limitations

41. TPG Transcription • C (Mm) my mom helped me in the bike. • E Tell me, keep going! • C Because I fall one day. • E Keep going! • C (And) and my mom say, “Just do it like this”. • C “And you go now”. • C (And) and my mom say, “Let’s do a race!” • C (And) and I won.

43. Benchmark Assessment (TNR) • Benchmark testing for all students • Use student’s best/median score out of 3 administrations • Use a criterion score (level of performance) to assign level of risk for each student (e.g., at-risk, some risk, no risk) • Use gain scores (or lack of gain) to identify students who would benefit from more intensive intervention

44. Progress Monitoring

47. What About Diagnosis? • Limited growth or response to instruction over time is a strong indicator of language impairment. • Chanthongthip, Petersen, Spencer, (2012)

49. What About Intervention Planning? • While some tools are not capable of informing instruction, the NLM measures can directly inform a child’s language needs. • Goals can be derived directly from the results of the NLM assessments. • Narratives …can be the intervention target …can be the intervention context