The Health Literacy Component (HLC) of the 2003 NAAL

1. A webcast presentation September 8, 2004 An Overview of the 2003 National Assessment of Adult Literacy (NAAL) Sheida White Project Officer National Assessment of Adult Literacy (NAAL) The Health Literacy Component (HLC) of the 2003 NAAL Sheida White Project Officer National Assessment of Adult Literacy (NAAL)

2. Topics covered by this presentation • Background, key features, and major goals • Performance of literacy tasks • Skills required to perform literacy tasks • Assessment design and administration • Data analysis and reporting

3. Background, Key Features, and Major Goals

4. NAAL background • In 1985 and 1992, the National Center for Education Statistics (NCES) conducted adult literacy assessments that focused on real-world tasks • The 2003 National Assessment of Adult Literacy (NAAL) builds on and expands previous NCES adult literacy assessments • NAAL is the first assessment of the nation’s progress in adult literacy since the 1992 National Adult Literacy Survey (NALS)

5. Key features of NAAL • Is based on in-person interviews of approximately 18,000 Americans age 16 and older • Provides data for the United States as a whole, individual states choosing to participate in a state-level assessment, and the nation’s prison population • Emphasizes the use of printed everyday materials (such as bills, prescriptions, newspapers) needed to function adequately in one’s environment • Provides data on background characteristics, performance of everyday literacy tasks, and skills underlying task performance

6. Major goals • Describe the status of adult literacy • Report on trends between 1992 and 2003 • Identify relationships between literacy and selected characteristics of adults • Provide new information about skills underlying adult literacy, including basic reading skills of the least-literate adults • Facilitate use of NAAL data by diverse audiences, including policymakers, researchers, and educators

7. Performance of Literacy Tasks

8. NAAL’s task-based literacy definition Literacy is the ability to use printed and writteninfor-mation [prose, document, and quantitative] to function in society, to achieve one’s goals, and to develop one’s knowledge and potential S. White 8-04

9. Definitions of three key literacy areas

10. Measurement of prose, document, and quantitative literacy • NAAL’s prose literacy, document literacy, and quantitative literacy scale scores are comparable to those of the 1992 NALS • Scores are based on completion of items that are all • Classified as prose, document, or quantitative • Open-ended to represent everyday tasks • Placed before stimulus materials to set a purpose for reading • Most items require • Short written responses • Searching the text for specific information

11. Topics of NAAL stimulus materials

12. Vitamin E (tocopherol)—helps protect red blood cells.May aid the circulatory system and counteract the agingprocess. Best sources: wheat germ, whole grains, eggs, pea-nuts, organ meats, margarine, vegetable oils, green leafy veg-etables. Sample prose item used in 1992 S. White 8-04

13. Sample document item used in 1992 S. White 8-04

14. S. White 8-04 Sample quantitative item used in 1992

15. A new health literacy score based on health-related items • NAAL includes 28 health-related items (out of 153 items) • All participants receive some health-related items • All health-related items are also classified as prose, document, or quantitative items • The health-related items • Are included with other items when calculating the prose, document, and quantitative scale scores • Are used to calculate a separate health literacy score, based solely on health-related items

16. Development and focus of the NAAL Health Literacy Component • NAAL’s Health Literacy Component was developed in response to a request by the U.S. Department of Health and Human Services (HHS) • NAAL focuses on one key aspect of health literacy—the ability to use printed and written materials to accomplish a broad range of health-related tasks • NAAL’s health-related tasks include • Clinical tasks, having to do with the health care itself • Preventive tasks, having to do with healthy habits and prevention of illness • Navigation tasks, having to do with bureaucratic demands to get to the health care

17. Skills Required to Perform Literacy Tasks

18. Purpose of the Adult Literacy Supplemental Assessment (ALSA) • The 1992 NALS provided little information about the literacy abilities of the least-literate adults, who were not able to complete the assessment • The purpose of the new Adult Literacy Supplemental Assessment (ALSA) is to get a clearer picture of the skills of the least-literate adults • ALSA is intended only for adults who would not be capable of meaningful participation in the “main NAAL” assessment, because they lack the skills to perform even the easiest tasks with a high degree of success • These adults take ALSA instead of the main NAAL

19. Selection of ALSA participants • All NAAL participants take seven easy core screening items • Interviewers score participants’ answers using the scoring guides provided on their computers • Computer selects ALSA participants using an empirically derived algorithm that predicts very low performance • The algorithm predicts which participants would get a total of five items or fewer correct (including the core items) • Five items, spread across the three NAAL scales, would not be sufficient to allow accurate estimates of performance on any of the scales

20. Important features of ALSA • ALSA instructions and responses are given orally and can be in either English or Spanish • ALSA materials are • Contextualized (with logos, pictures, etc.) • Familiar (as measured by background questions) • Tangible (e.g., food boxes, medicine bottles, utility bills) • ALSA tasks, although simple and contextualized, still require participants to read at least parts of the words

21. The need for additional data about skill deficits • In addition to the 6% of adults who lacked the skills to complete the 1992 NALS, an additional 15% of adults were able to perform only the easiest NALS tasks (classified as “Level 1” tasks) • For adults with low scores, more information is needed about the nature of the skill deficits • Lack of basic (word-level) reading skills? • Lack of fluency in basic skills? • Lack of higher level literacy skills?

22. Range of skill deficits and their potential implications

23. New data on the skills of all adults • The Fluency Addition to NAAL (FAN) provides new data on the skills of all adults • After completing either the main NAAL or ALSA, all participants take FAN, which • Consists of timed oral reading tasks • Describes the oral reading skills of adults (e.g., words read correctly per minute) • Provides a measure of basic (word-level) reading skills • FAN scores can be compared with main NAAL scores at various performance levels, ALSA scores, and health literacy scores

24. FAN oral reading materials and associated measures • Pseudoword lists, consisting of possible but nonoccurring English forms (e.g., “wike”), provide a measure of adults’ ability to “decode” (or identify the sounds of) words with which they are not familiar • Word lists, consisting of English words arranged in increasing order of difficulty, provide a measure of adults’ ability to recognize familiar words (often referred to as “sight words”) as well as to decode • Text passages, consisting of 150–200 words each, provide a measure of adults’ ability to read words in connected texts

25. Initial measures and scoring challenges for FAN • Initial FAN target measures are • Total words read aloud—whether correctly or not—per unit of time* • Words read correctly per unit of time* • Words read correctly as a percentage of total words read • Challenges in scoring FAN data include ensuring • That correctness can be measured reliably • That speakers of nonstandard varieties of English are not unfairly penalized *Unit of time: 60 seconds for text passages or 20 seconds for lists

26. Key similarities and differences between FAN and ALSA • Similarities include the following: • Both measure basic reading skills • Both have oral instructions in either English or Spanish • Both require reading words and connected text • Differences include the following: • Only ALSA permits the use of compensatory strategies, while FAN does not provide any nonlinguistic clues (such as pictures) • ALSA connected texts are fewer, easier, and shorter (12 sentences each in ALSA, compared with 150200 words each in FAN)

27. The functionality of word-level skills and higher level literacy skills • NAAL defines literacy as the ability to use printed and written information • As defined by NAAL, all literacy is functional; therefore, NAAL does not differentiate between “literacy” and “functional literacy” • Certain skills, including word-level reading skills, are needed to successfully perform NAAL tasks • The ability to read words is a functional skill, but not the only one • In addition to word-level skills, NAAL recognizes six types of literacy skills

28. The six types of literacy skills NAAL literacy skills Language comprehension skills Text search skills Inferential skills Understanding the structure and meaning of sentences Drawing appropriate text-based inferences Searching textefficiently • Computation performance skills • Computation identification skills • Application skills Identifying the calculations required to solve quantitative problems • Performing any required calculations • Using newly searched, inferred, or computed information to accomplish a variety of goals

29. Analysis of main NAAL data in terms of underlying skills Recently developed multidimensional analysis methods will be applied to 2003 main NAAL data in order to • Characterize the proficiency of adults in terms of the NAAL literacy skills • Characterize effects of task demands and written materials on the difficulty of literacy tasks

30. Assessment Design and Administration

31. Selection of the national sample for NAAL

32. Key features of the NAAL sample • Adults age 16 and over living primarily in households • Nationally representative sample of 10,006 adults, augmented by • Samples from the six states that participated at the state level (an additional 7,166 adults) • Prison sample of inmates (about 1,000 more adults) • Incentive payment of $30 to increase representativeness of the sample and response rate • Oversampling of Blacks and Hispanics

33. Matrix sampling • The main NAAL has a total of 153 items (including the 7 core screening items that are given to all participants and used to select ALSA participants) • Matrix sampling—administering only a portion of the non-core items to each respondent—ensures broad coverage while limiting respondent burden • Each respondent takes 1 of 26 booklets • Each booklet includes 3 of 13 blocks of items (for about 40 items per booklet, including the core items) • Each block is spiraled in the 26 booklets (i.e., paired with every other block)

34. 6 from 1985 7 new for 1992 6 from 1992 7 new for 2003 Allowing comparisons with the 1992 NALS while introducing new features 1992 assessment 2003 assessment Six of the blocks from 1992 are used again in 2003. Seven new (2003) blocks replace all six of the blocks from 1985 plus one of the blocks from 1992. 13 blocks total 13 blocks total S. White 8-04

35. Allowing comparisons with the 1992 NALS while introducing new features (continued) • The seven blocks that were newly created for 2003 • Are similar to the 1992 blocks in terms of the distribution of skills required by tasks • Have approximately the same average task difficulty as the 1992 blocks • Are spiraled with the 1992 blocks in the 26 booklets • Item response theory will be used to link the 1992 and 2003 scales using the tasks common to both assessments

36. New performance levels for 2003 • In 1992, five performance levels were developed by grouping together tasks of similar difficulty • The 1992 levels proved to have some limitations • Scores that are the cut points between levels vary depending on the measure of task difficulty used • No distinctions are made within the lowest level of performance (Level 1) • For 2003, new performance levels are being developed for the main NAAL by the National Academy of Sciences • Goals of the new levels include increased clarity and better differentiation among adults at the lower end of the scale

37. NAAL total average assessment time Main NAALALSA Background questionnaire 28 min 28 min Core screening items 8 min 8 min Main NAAL or ALSA assessment 39 min 25 min FAN 14 min12 min Total 89 min 73 min • Respondents may take as long as they need to perform main NAAL or ALSA tasks, but are encouraged to move to a new task when stuck or frustrated

38. Accommodations for adults with special needs • The assessment is conducted in the participant’s home • The assessment is administered one on one • All participants receive additional time to complete the main NAAL or ALSA if they need it • The background questionnaire is administered orally in either English or Spanish • Instructions for the core screening items and for ALSA are given orally in either English or Spanish • Participants with a native language other than English or Spanish may take the core screening items even if they cannot complete the background questionnaire

39. Expanded NAAL background questionnaire For 2003, the background questionnaire was expanded to better reflect • Economic and technological developments in the past decade • The background of low-performing adults • Health-related activities of adults • Literacy-related activities of parents and their children

40. Characteristics of the NAAL scoring rubrics: Statistical considerations Responses to each item are analyzed for • Frequency with which a particular response is given • Interrater reliability in scoring responses • P-value (percentage of the population responding correctly) • Biserial (and polyserial) correlation between each response (and partially correct response) and the respondent’s overall score • Range findings for delineation of the range of responses received • Differential item functioning (DIF) in favor of certain groups

41. Development of the NAAL scoring rubrics:Content considerations • Seek evidence that adults can indeed use printed materials • Allow for partially correct responses if the information provided is still useful in accomplishing the task • Score most responses as either correct or incorrect, since often a partially completed task is not an accomplished task • Allow for writing errors as long as the overall meaning of the response is correct • Distinguish responses providing specific information from those providing generalizations

42. Data Analysis and Reporting

43. - New software for data analysis • AM software has been developed by the American Institutes for Research (AIR), with NCES funding • The new software will reduce the amount of time required to analyze data by several months • The software will allow the technical report and the Data File Users Manual to be produced on a more timely schedule than in 1992 • The software will also make it easier for states and researchers to do their own data analyses to supplement the information in the state and national reports

44. Average scores and distributions Average scale scores and score distributions will be provided • For the population as a whole • For population groups • But notfor individuals

45. Inappropriateness of individual scores • No respondent takes all of the assessment • It would not be appropriate to calculate individual scores based on part of the assessment because • The individual scores would be unreliable (a reliable or consistent measure requires many items) • The individual scores would not be a valid representa-tion of the domain of adult literacy (valid representation requires many types of tasks, genres, etc.)

46. Trend analysis Results from the 1992 NALS and the 2003 main NAAL will be compared in terms of • Average scale scores for prose, document, and quantitative literacy • Performance levels set by the National Academy of Sciences • P-values (the percentage of the population giving the correct answer to each item)

47. New release of items and p-values • About 90 assessment items used in 1992 assessment will soon be released • All released items will be available on the NAAL website (http://nces.ed.gov/naal) • Each item’s p-value will be published • This will be the first publication of p-values for 1992 assessment items • P-values will be published for all items used in 1992—even items that are not released

48. Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1

49. Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #2

50. New types of results in 2003 • Health literacy scale scores measuring the performance of American adults on health-related tasks • Skill-based analyses inferring the literacy skills associated with the ability to perform main NAAL tasks • ALSA scores providing information on the literacy skills and deficits of America’s least-literate adults • FAN oral reading and basic skills data providing information about the skills of all adults