Comparative Measurement of Higher Education Learning: Lessons from Collegiate Learning Assessment Study

1. Models for Future Comparative Measurement of Higher Education Learning: Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.* Richard Arum New York University and Social Science Research Council * Josipa Roksa (University of Virginia) and Melissa Velez (NYU) collaborated on research findings presented here. We thank Ford and Lumina Foundations for their generous financial support and the Council for Aid to Education for assistance with data collection.

2. College Learning in the Spotlight (U.S. Policy Context) “As other nations rapidly improve their higher education systems, we are disturbed by evidence that the quality of student learning at U.S. colleges and universities is inadequate, and in some cases, declining.” A Test of Leadership U.S. Secretary of Education’s Commission on the Future of Higher Education (2006)

3. College Learning in the Spotlight (U.S. Policy Context) “These shortcomings have real-world consequences. Employers report repeatedly that many new graduates they hire are not prepared to work, lacking the critical thinking, writing and problem-solving skills needed in today’s workplaces.” A Test of Leadership U.S. Secretary of Education’s Commission on the Future of Higher Education (2006)

4. Measurement of Learning in U.S. Higher Education Dearth of direct measures of higher education student learning that are comparable across institutions and/or states Measuring Up 2008 – Assigned a grade of Incomplete to all states in the area of measuring learning: “All states receive an ‘incomplete’ in learning because there are not sufficient data to allow meaningful state-by-state comparisons.”

5. Measurement Challenges • Curriculum varies widely across fields of study and institutions – little consensus on what is to be learned • Practitioner resistance to “reductionist” approaches • Students are sorted by ability and other factors into different institutions

6. Collegiate Learning Assessment (CLA) Dimensions of learning assessed critical thinking, analytical reasoning, and written communication Distinguishing characteristics Direct measures (as opposed to student reports) NOT multiple choice Holistic assessment based on open-ended prompts representing “real-world” scenarios

7. Collegiate Learning Assessment (CLA) Components Performance task Make an argument Break an argument

8. Performance Task (example) You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech’s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235.

9. Performance Task (example, cont.) Students are provided with a set of materials (e.g. newspaper articles, Federal Accident Report, e-mail exchanges, description and performance characteristics of AirSwift 235 and another model, etc.) and asked to prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors may have contributed to the accident and should be taken into account, and their overall recommendation about whether or not DynaTech should purchase the plane.

10. Determinants of College Learning Dataset Longitudinal Design Fall 2005 and Spring 2007 (beginning of freshman and end of sophomore years) Large Scale 24 diverse four-year institutions; 2,341 students Breath of Information Family background and high school information, college experiences and contexts, college transcripts Collegiate Learning Assessment (CLA)

11. Sample Characteristics: Who are These Students?

12. Research Questions What individual, social and institutional factors are associated with learning in higher education? How do disadvantaged groups of students fare in college in terms of measured learning? To what extent do individual, social and institutional factors account for variation across disadvantaged groups?

13. Overview of the Conceptual Model Employed in the Study Measures of Disadvantage: Race/Ethnicity Parental Education Parental Occupation Racially Segregated High School (70+ % minority) Non-English Language Control Variables: 2005 Test Score Gender Two Parent Household Sibling Number Urbanicity Geographic Region 2007 Test Score High School Academic Preparation: GPANumber of AP Courses Taken College Experiences: Hours Spent Studying Alone Hours Spent Studying with Peers Hours Spent in a Fraternity/Sorority Hours Worked On/Off Campus Faculty Expectations Field of Study College Fixed Effects

14. Analysis - Part I Individual, Social and Institutional Factors Associated with Learning as Measured by Improvement in CLA Performance

15. High School Preparation Figure 1. Predicted 2007 Test Score by Number of High School AP Courses

16. College Engagement and Learning Figure 2. Predicted 2007 Score by College Engagement and Involvement Measures

17. College Employment and Learning Figure 3. Predicted 2007 Test Score by Employment Measures

18. Faculty Expectations and Learning Figure 4. Predicted 2007 Test Score by Level of Faculty Expectations

19. Fields of Study and Learning Figure 5. Predicted 2007 Test Score by College Major

20. Analysis - Part II Social Disadvantaged Group Differences in Learning as Measured by Improvement in CLA Performance

21. CLA Performance by Race Figure 6. 2005 and 2007 Test Scores by Race Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

22. CLA Performance by Parental Education Figure 7. 2005 and 2007 Test Scores by Parental Education Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

23. CLA Performance by High School Student Composition and Home Language Figure 8. 2005 and 2007 Test Scores by Level of High School Student Composition and Home Language Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

24. Analysis - Part III Accounting for Variation in CLA Performance by Social Disadvantaged Groups

25. Accounting for Group Differences: H.S.-College Experiences and Institutional Differences Figure 11. Test score gaps in baseline and full models with college institutional fixed effects. Note: Baseline regression model predicts the 2007 score, controlling for the 2005 score and a range of background characteristics. Full model also includes measures of high school academic preparation and college experiences. Non-significant differences are shaded.

26. Conclusions and Implications Policy makers need to focus attention on improving individual student learning in higher education, not just access and retention. Practitioners need to recognize the extent to which both student experiences as well as institutional differences are associated with variation in learning. Additional systematic longitudinal research is necessary to improve understanding of these processes. Measurement of learning across fields and institutions is possible with instruments such as the CLA.