IALS Measurement Validity, Program Impact and Logic Models for Policy Development

1. IALS Measurement Validity, Program Impact and Logic Models for Policy Development Stephen Reder Portland State University I The Centre for Literacy Fall Institute Banff, Alberta – October 25, 2011 IALS, Its Meaning and Impact for Policy and Practice

2. The Longitudinal Study of Adult Learning (LSAL) Portland State University funded by U.S. Department of Education and National Institute for Literacy

3. Periods of Participation in Basic Skills Programs Since Leaving School

4. Self-Study in the LSAL • Defined as “studying on your own to improve your reading, writing or math skills or prepare for the GED” • Probes distinguished such self-study from activity conducted in school or as part of a basic skills or GED class • In-depth qualitative interviews confirmed the validity of these self-reports

5. Self-Study by Literacy Proficiency(wave 1)

6. Self-Study: Skills “Practiced”(Wave 3)

7. Self-Study: Materials Used(Wave 3) By Wave 5, computer use for self-study rose to 51%

8. Self-Study and Participation 64% have self-studied self-study only neither adult ed only 60% have taken adult ed self-study & adult ed (between leaving school and Wave 4)

9. Self-Reported Change in Literacy Practices by Participation & Self-Study

10. Percent GED Attainment by Participation & Self-Study

11. Some Implications • We need a broader construct of participation that • Includes programs, self-study & other learning activities • Sees self-study in a continuum of provision strategies along with various types of classroom and tutoring programs • Conceives of learners as actively deploying resources as well as programs delivering services

12. Some Implications (con’t) • We need broader programmatic structures that support & coordinate different modes of participation • Technologies may be particularly helpful here but need to provide more than just direct instruction (e.g., support goal setting & plan development, counselling, etc.)

13. Modeling Lifespan Literacy Development • Program participation and self-study have positive, short-term effects on engagement in literacy practices • Engagement in literacy practices over time has cumulative effects on the growth of literacyproficiency • This leads to a broader concept of participation that involes both programs and self-study • Self-study bridges between periods of program participation and helps broaden persistence of attendance into persistence of learning

14. Programs as Practice-BuildingContexts Programs that build engagement in authentic literacy practices may lead to long-term proficiency growth that is not evident in short-term pre-post testing

15. Programs as Facilitated LearningContexts • Programs often blend time in classroom instruction with time in learning centers • There may be a continuum of facilitated learning with classroom activities on one end and self-study on the other end, with facilitated use of learning centers and tutoring in-between

16. Redefining Program, Participation and Persistence • Focus on engagement in literacy practices rather than on persistence in programs • Broaden definition of program to include facilitated engagement in literacy practices • Technology has broader role here than just delivering distance learning

17. Effects of Instruction • Strongest evidence of program impact: Controlled comparison of participants’ and non-participants’ gains • Next strongest: Covariation of hours of instruction with observed gains • My analysis of CASAS proficiency gains in several states’ NRS data indicates the gains attributable to instructional hours are greatly attenuated (if present at all) when test experience is controlled

18. Program Participation.. • ..is often fragmented by life circumstances • ..is better thought of as a busy intersection among life and learning trajectories rather than as a parking lot • ..should support students’ trajectories not just their activities in the intersections • ..is expanded by a learning plan that learners can follow and that services can be wrapped around

19. Learning Support Systems • Adults need learning support systems that provide portable, personalized learning plans they can follow to reach their goals • Learning support systems include a focus on building engagement in literacy practices • Learning support systems provide flexible blends of online and offline local resources to support learning • The technology supports learners directly as they follow a learning plan and indirectly by coordinating the work of those who support them • Communities design and implement local learning support systems, utilizing the technology to facilitate collaboration and information-sharing among the various organizations working with learners

20. A New Logic Model for Program Impact • Practice-engagement theory • Busy intersection vs. parking lot • Learning support system

21. Research Recommendations • Improve ways of measuring engagement in literacy practices • Conduct interventions that stimulate increased practice-engagement • Need long-term longitudinal studies with multiple repeated literacy outcome measures • Note that National Reading Panel found little evidence indicating that programs which aim to increase independent reading in schoolchildren lead to increased proficiency • Examine literacy changes (including proficiency loss) in older adults and interventions to maintain literacy capabilities in older adults

22. Contact information Steve Reder Professor, Chair – Department of Applied Linguistics Portland State University (503) 725-3999 reders@pdx.edu For more information and references to the research covered in the Powerpoint presentation, please see: Longitudinal Study of Adult Learning www.lsal.pdx.edu Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development www.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdf