A webcast presentation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 52

The Health Literacy Component (HLC) of the 2003 NAAL PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on
  • Presentation posted in: General

A webcast presentation September 8, 2004. An Overview of the 2003 National Assessment of Adult Literacy (NAAL). Sheida White Project Officer National Assessment of Adult Literacy (NAAL). The Health Literacy Component (HLC) of the 2003 NAAL. Sheida White

Download Presentation

The Health Literacy Component (HLC) of the 2003 NAAL

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The health literacy component hlc of the 2003 naal

A webcast presentation

September 8, 2004

An Overview of the 2003 National Assessment of Adult Literacy (NAAL)

Sheida White

Project Officer National Assessment of Adult Literacy

(NAAL)

The Health Literacy Component (HLC)

of the 2003 NAAL

Sheida White

Project Officer National Assessment of Adult Literacy

(NAAL)


The health literacy component hlc of the 2003 naal

Topics covered by this presentation

  • Background, key features, and major goals

  • Performance of literacy tasks

  • Skills required to perform literacy tasks

  • Assessment design and administration

  • Data analysis and reporting


The health literacy component hlc of the 2003 naal

Background, Key Features, and Major Goals


The health literacy component hlc of the 2003 naal

NAAL background

  • In 1985 and 1992, the National Center for Education Statistics (NCES) conducted adult literacy assessments that focused on real-world tasks

  • The 2003 National Assessment of Adult Literacy (NAAL) builds on and expands previous NCES adult literacy assessments

  • NAAL is the first assessment of the nation’s progress in adult literacy since the 1992 National Adult Literacy Survey (NALS)


The health literacy component hlc of the 2003 naal

Key features of NAAL

  • Is based on in-person interviews of approximately 18,000 Americans age 16 and older

  • Provides data for the United States as a whole, individual states choosing to participate in a state-level assessment, and the nation’s prison population

  • Emphasizes the use of printed everyday materials (such as bills, prescriptions, newspapers) needed to function adequately in one’s environment

  • Provides data on background characteristics, performance of everyday literacy tasks, and skills underlying task performance


The health literacy component hlc of the 2003 naal

Major goals

  • Describe the status of adult literacy

  • Report on trends between 1992 and 2003

  • Identify relationships between literacy and selected characteristics of adults

  • Provide new information about skills underlying adult literacy, including basic reading skills of the least-literate adults

  • Facilitate use of NAAL data by diverse audiences, including policymakers, researchers, and educators


The health literacy component hlc of the 2003 naal

Performance of Literacy Tasks


The health literacy component hlc of the 2003 naal

NAAL’s task-based literacy definition

Literacy is the ability to use printed and writteninfor-mation [prose, document, and quantitative] to function in society, to achieve one’s goals, and to develop one’s knowledge and potential

S. White 8-04


The health literacy component hlc of the 2003 naal

Definitions of three key literacy areas


The health literacy component hlc of the 2003 naal

Measurement of prose, document, and quantitative literacy

  • NAAL’s prose literacy, document literacy, and quantitative literacy scale scores are comparable to those of the 1992 NALS

  • Scores are based on completion of items that are all

    • Classified as prose, document, or quantitative

    • Open-ended to represent everyday tasks

    • Placed before stimulus materials to set a purpose for reading

  • Most items require

    • Short written responses

    • Searching the text for specific information


The health literacy component hlc of the 2003 naal

Topics of NAAL stimulus materials


The health literacy component hlc of the 2003 naal

Vitamin E (tocopherol)—helps protect red blood cells.May aid the circulatory system and counteract the agingprocess. Best sources: wheat germ, whole grains, eggs, pea-nuts, organ meats, margarine, vegetable oils, green leafy veg-etables.

Sample prose item used in 1992

S. White 8-04


The health literacy component hlc of the 2003 naal

Sample document item used in 1992

S. White 8-04


The health literacy component hlc of the 2003 naal

S. White 8-04

Sample quantitative item used in 1992


The health literacy component hlc of the 2003 naal

A new health literacy score based on health-related items

  • NAAL includes 28 health-related items (out of 153 items)

  • All participants receive some health-related items

  • All health-related items are also classified as prose, document, or quantitative items

  • The health-related items

    • Are included with other items when calculating the prose, document, and quantitative scale scores

    • Are used to calculate a separate health literacy score, based solely on health-related items


The health literacy component hlc of the 2003 naal

Development and focus of the NAAL Health Literacy Component

  • NAAL’s Health Literacy Component was developed in response to a request by the U.S. Department of Health and Human Services (HHS)

  • NAAL focuses on one key aspect of health literacy—the ability to use printed and written materials to accomplish a broad range of health-related tasks

  • NAAL’s health-related tasks include

    • Clinical tasks, having to do with the health care itself

    • Preventive tasks, having to do with healthy habits and prevention of illness

    • Navigation tasks, having to do with bureaucratic demands to get to the health care


The health literacy component hlc of the 2003 naal

Skills Required to Perform Literacy Tasks


The health literacy component hlc of the 2003 naal

Purpose of the Adult Literacy Supplemental Assessment (ALSA)

  • The 1992 NALS provided little information about the literacy abilities of the least-literate adults, who were not able to complete the assessment

  • The purpose of the new Adult Literacy Supplemental Assessment (ALSA) is to get a clearer picture of the skills of the least-literate adults

  • ALSA is intended only for adults who would not be capable of meaningful participation in the “main NAAL” assessment, because they lack the skills to perform even the easiest tasks with a high degree of success

  • These adults take ALSA instead of the main NAAL


The health literacy component hlc of the 2003 naal

Selection of ALSA participants

  • All NAAL participants take seven easy core screening items

  • Interviewers score participants’ answers using the scoring guides provided on their computers

  • Computer selects ALSA participants using an empirically derived algorithm that predicts very low performance

  • The algorithm predicts which participants would get a total of five items or fewer correct (including the core items)

  • Five items, spread across the three NAAL scales, would not be sufficient to allow accurate estimates of performance on any of the scales


The health literacy component hlc of the 2003 naal

Important features of ALSA

  • ALSA instructions and responses are given orally and can be in either English or Spanish

  • ALSA materials are

    • Contextualized (with logos, pictures, etc.)

    • Familiar (as measured by background questions)

    • Tangible (e.g., food boxes, medicine bottles, utility bills)

  • ALSA tasks, although simple and contextualized, still require participants to read at least parts of the words


The health literacy component hlc of the 2003 naal

The need for additional data about skill deficits

  • In addition to the 6% of adults who lacked the skills to complete the 1992 NALS, an additional 15% of adults were able to perform only the easiest NALS tasks (classified as “Level 1” tasks)

  • For adults with low scores, more information is needed about the nature of the skill deficits

    • Lack of basic (word-level) reading skills?

    • Lack of fluency in basic skills?

    • Lack of higher level literacy skills?


The health literacy component hlc of the 2003 naal

Range of skill deficits and their potential implications


The health literacy component hlc of the 2003 naal

New data on the skills of all adults

  • The Fluency Addition to NAAL (FAN) provides new data on the skills of all adults

  • After completing either the main NAAL or ALSA, all participants take FAN, which

    • Consists of timed oral reading tasks

    • Describes the oral reading skills of adults (e.g., words read correctly per minute)

    • Provides a measure of basic (word-level) reading skills

  • FAN scores can be compared with main NAAL scores at various performance levels, ALSA scores, and health literacy scores


The health literacy component hlc of the 2003 naal

FAN oral reading materials and associated measures

  • Pseudoword lists, consisting of possible but nonoccurring English forms (e.g., “wike”), provide a measure of adults’ ability to “decode” (or identify the sounds of) words with which they are not familiar

  • Word lists, consisting of English words arranged in increasing order of difficulty, provide a measure of adults’ ability to recognize familiar words (often referred to as “sight words”) as well as to decode

  • Text passages, consisting of 150–200 words each, provide a measure of adults’ ability to read words in connected texts


The health literacy component hlc of the 2003 naal

Initial measures and scoring challenges for FAN

  • Initial FAN target measures are

    • Total words read aloud—whether correctly or not—per unit of time*

    • Words read correctly per unit of time*

    • Words read correctly as a percentage of total words read

  • Challenges in scoring FAN data include ensuring

    • That correctness can be measured reliably

    • That speakers of nonstandard varieties of English are not unfairly penalized

      *Unit of time: 60 seconds for text passages or 20 seconds for lists


The health literacy component hlc of the 2003 naal

Key similarities and differences between FAN and ALSA

  • Similarities include the following:

    • Both measure basic reading skills

    • Both have oral instructions in either English or Spanish

    • Both require reading words and connected text

  • Differences include the following:

    • Only ALSA permits the use of compensatory strategies, while FAN does not provide any nonlinguistic clues (such as pictures)

    • ALSA connected texts are fewer, easier, and shorter (12 sentences each in ALSA, compared with 150200 words each in FAN)


The health literacy component hlc of the 2003 naal

The functionality of word-level skills and higher level literacy skills

  • NAAL defines literacy as the ability to use printed and written information

  • As defined by NAAL, all literacy is functional; therefore, NAAL does not differentiate between “literacy” and “functional literacy”

  • Certain skills, including word-level reading skills, are needed to successfully perform NAAL tasks

  • The ability to read words is a functional skill, but not the only one

  • In addition to word-level skills, NAAL recognizes six types of literacy skills


The six types of literacy skills

The six types of literacy skills

NAAL literacy skills

Language comprehension skills

Text search skills

Inferential skills

Understanding the structure and meaning of sentences

Drawing appropriate text-based inferences

Searching textefficiently

  • Computation performance skills

  • Computation identification skills

  • Application skills

Identifying the calculations required to solve quantitative problems

  • Performing any required calculations

  • Using newly searched, inferred, or computed information to accomplish a variety of goals


The health literacy component hlc of the 2003 naal

Analysis of main NAAL data in terms of underlying skills

Recently developed multidimensional analysis methods will be applied to 2003 main NAAL data in order to

  • Characterize the proficiency of adults in terms of the NAAL literacy skills

  • Characterize effects of task demands and written materials on the difficulty of literacy tasks


The health literacy component hlc of the 2003 naal

Assessment Design and Administration


The health literacy component hlc of the 2003 naal

Selection of the national sample for NAAL


The health literacy component hlc of the 2003 naal

Key features of the NAAL sample

  • Adults age 16 and over living primarily in households

  • Nationally representative sample of 10,006 adults, augmented by

    • Samples from the six states that participated at the state level (an additional 7,166 adults)

    • Prison sample of inmates (about 1,000 more adults)

  • Incentive payment of $30 to increase representativeness of the sample and response rate

  • Oversampling of Blacks and Hispanics


The health literacy component hlc of the 2003 naal

Matrix sampling

  • The main NAAL has a total of 153 items (including the 7 core screening items that are given to all participants and used to select ALSA participants)

  • Matrix sampling—administering only a portion of the non-core items to each respondent—ensures broad coverage while limiting respondent burden

    • Each respondent takes 1 of 26 booklets

    • Each booklet includes 3 of 13 blocks of items (for about 40 items per booklet, including the core items)

    • Each block is spiraled in the 26 booklets (i.e., paired with every other block)


The health literacy component hlc of the 2003 naal

6 from 1985

7 new for 1992

6 from 1992

7 new for 2003

Allowing comparisons with the 1992 NALS while introducing new features

1992 assessment

2003 assessment

Six of the blocks from 1992 are used again in 2003.

Seven new (2003) blocks replace all six of the blocks from 1985 plus one of the blocks from 1992.

13 blocks total

13 blocks total

S. White 8-04


The health literacy component hlc of the 2003 naal

Allowing comparisons with the 1992 NALS while introducing new features (continued)

  • The seven blocks that were newly created for 2003

    • Are similar to the 1992 blocks in terms of the distribution of skills required by tasks

    • Have approximately the same average task difficulty as the 1992 blocks

    • Are spiraled with the 1992 blocks in the 26 booklets

  • Item response theory will be used to link the 1992 and 2003 scales using the tasks common to both assessments


The health literacy component hlc of the 2003 naal

New performance levels for 2003

  • In 1992, five performance levels were developed by grouping together tasks of similar difficulty

  • The 1992 levels proved to have some limitations

    • Scores that are the cut points between levels vary depending on the measure of task difficulty used

    • No distinctions are made within the lowest level of performance (Level 1)

  • For 2003, new performance levels are being developed for the main NAAL by the National Academy of Sciences

  • Goals of the new levels include increased clarity and better differentiation among adults at the lower end of the scale


The health literacy component hlc of the 2003 naal

NAAL total average assessment time

Main NAALALSA

Background questionnaire28 min28 min

Core screening items 8 min 8 min

Main NAAL or ALSA assessment39 min25 min

FAN14 min12 min

Total89 min73 min

  • Respondents may take as long as they need to perform main NAAL or ALSA tasks, but are encouraged to move to a new task when stuck or frustrated


The health literacy component hlc of the 2003 naal

Accommodations for adults with special needs

  • The assessment is conducted in the participant’s home

  • The assessment is administered one on one

  • All participants receive additional time to complete the main NAAL or ALSA if they need it

  • The background questionnaire is administered orally in either English or Spanish

  • Instructions for the core screening items and for ALSA are given orally in either English or Spanish

  • Participants with a native language other than English or Spanish may take the core screening items even if they cannot complete the background questionnaire


The health literacy component hlc of the 2003 naal

Expanded NAAL background questionnaire

For 2003, the background questionnaire was expanded to better reflect

  • Economic and technological developments in the past decade

  • The background of low-performing adults

  • Health-related activities of adults

  • Literacy-related activities of parents and their children


The health literacy component hlc of the 2003 naal

Characteristics of the NAAL scoring rubrics:

Statistical considerations

Responses to each item are analyzed for

  • Frequency with which a particular response is given

  • Interrater reliability in scoring responses

  • P-value (percentage of the population responding correctly)

  • Biserial (and polyserial) correlation between each response (and partially correct response) and the respondent’s overall score

  • Range findings for delineation of the range of responses received

  • Differential item functioning (DIF) in favor of certain groups


The health literacy component hlc of the 2003 naal

Development of the NAAL scoring rubrics:Content considerations

  • Seek evidence that adults can indeed use printed materials

  • Allow for partially correct responses if the information provided is still useful in accomplishing the task

  • Score most responses as either correct or incorrect, since often a partially completed task is not an accomplished task

  • Allow for writing errors as long as the overall meaning of the response is correct

  • Distinguish responses providing specific information from those providing generalizations


The health literacy component hlc of the 2003 naal

Data Analysis and Reporting


The health literacy component hlc of the 2003 naal

-

New software for data analysis

  • AM software has been developed by the American Institutes for Research (AIR), with NCES funding

  • The new software will reduce the amount of time required to analyze data by several months

  • The software will allow the technical report and the Data File Users Manual to be produced on a more timely schedule than in 1992

  • The software will also make it easier for states and researchers to do their own data analyses to supplement the information in the state and national reports


The health literacy component hlc of the 2003 naal

Average scores and distributions

Average scale scores and score distributions will be

provided

  • For the population as a whole

  • For population groups

  • But notfor individuals


The health literacy component hlc of the 2003 naal

Inappropriateness of individual scores

  • No respondent takes all of the assessment

  • It would not be appropriate to calculate individual scores based on part of the assessment because

    • The individual scores would be unreliable (a reliable or consistent measure requires many items)

    • The individual scores would not be a valid representa-tion of the domain of adult literacy (valid representation requires many types of tasks, genres, etc.)


The health literacy component hlc of the 2003 naal

Trend analysis

Results from the 1992 NALS and the 2003 main NAAL will be compared in terms of

  • Average scale scores for prose, document, and quantitative literacy

  • Performance levels set by the National Academy of Sciences

  • P-values (the percentage of the population giving the correct answer to each item)


The health literacy component hlc of the 2003 naal

New release of items and p-values

  • About 90 assessment items used in 1992 assessment will soon be released

  • All released items will be available on the NAAL website (http://nces.ed.gov/naal)

  • Each item’s p-value will be published

  • This will be the first publication of p-values for 1992 assessment items

  • P-values will be published for all items used in 1992—even items that are not released


The health literacy component hlc of the 2003 naal

Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1


The health literacy component hlc of the 2003 naal

Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #2


The health literacy component hlc of the 2003 naal

New types of results in 2003

  • Health literacy scale scores measuring the performance of American adults on health-related tasks

  • Skill-based analyses inferring the literacy skills associated with the ability to perform main NAAL tasks

  • ALSA scores providing information on the literacy skills and deficits of America’s least-literate adults

  • FAN oral reading and basic skills data providing information about the skills of all adults


The health literacy component hlc of the 2003 naal

NAAL mailing list

  • If you would like to receive NAAL information tailored to your interests, simply fill out a short online form at http://www.edpubs.org/list/NAAL

  • On the form, you can indicate up to 16 areas of interest

    • Adult basic education

    • Adult secondary education

    • Citizenship/civics education

    • College prep

    • Computer literacy

    • Correctional education

    • Early childhood/children’s literacy

    • ESL/ESOL (English as a second language)

  • Family literacy

  • Health literacy

  • Homeless education

  • Learning disabilities

  • Mathematical literacy

  • Native language literacy (i.e., literacy in Spanish)

  • Policy and legislation

  • Workplace literacy


The health literacy component hlc of the 2003 naal

National Center for Education Statistics

Sheida WhiteNAAL Project Officer(202) [email protected]

Andrew KolstadSenior Technical Adviser(202) [email protected]

Websitehttp://nces.ed.gov/naal


  • Login