richard arum new york university and social science research council n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Richard Arum New York University and Social Science Research Council PowerPoint Presentation
Download Presentation
Richard Arum New York University and Social Science Research Council

Loading in 2 Seconds...

play fullscreen
1 / 26

Richard Arum New York University and Social Science Research Council - PowerPoint PPT Presentation


  • 168 Views
  • Uploaded on

Models for Future Comparative Measurement of Higher Education Learning: Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.*. Richard Arum New York University and Social Science Research Council.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Richard Arum New York University and Social Science Research Council' - midori


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
richard arum new york university and social science research council

Models for Future Comparative Measurement of Higher Education Learning: Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.*

Richard Arum

New York University and

Social Science Research Council

* Josipa Roksa (University of Virginia) and Melissa Velez (NYU) collaborated on research findings presented here. We thank Ford and Lumina Foundations for their generous financial support and the Council for Aid to Education for assistance with data collection.

college learning in the spotlight u s policy context
College Learning in the Spotlight (U.S. Policy Context)

“As other nations rapidly improve their higher education systems, we are disturbed by evidence that the quality of student learning at U.S. colleges and universities is inadequate, and in some cases, declining.”

A Test of Leadership

U.S. Secretary of Education’s Commission

on the Future of Higher Education (2006)

college learning in the spotlight u s policy context1
College Learning in the Spotlight (U.S. Policy Context)

“These shortcomings have real-world consequences. Employers report repeatedly that many new graduates they hire are not prepared to work, lacking the critical thinking, writing and problem-solving skills needed in today’s workplaces.”

A Test of Leadership

U.S. Secretary of Education’s Commission

on the Future of Higher Education (2006)

measurement of learning in u s higher education
Measurement of Learning in U.S. Higher Education

Dearth of direct measures of higher education student learning that are comparable across institutions and/or states

Measuring Up 2008 – Assigned a grade of Incomplete to all states in the area of measuring learning: “All states receive an ‘incomplete’ in learning because there are not sufficient data to allow meaningful state-by-state comparisons.”

measurement challenges
Measurement Challenges
  • Curriculum varies widely across fields of study and institutions – little consensus on what is to be learned
  • Practitioner resistance to “reductionist” approaches
  • Students are sorted by ability and other factors into different institutions
collegiate learning assessment cla
Collegiate Learning Assessment (CLA)

Dimensions of learning assessed

critical thinking, analytical reasoning, and written communication

Distinguishing characteristics

Direct measures (as opposed to student reports)

NOT multiple choice

Holistic assessment based on open-ended prompts representing “real-world” scenarios

collegiate learning assessment cla1
Collegiate Learning Assessment (CLA)

Components

Performance task

Make an argument

Break an argument

performance task example
Performance Task (example)

You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech’s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235.

performance task example cont
Performance Task (example, cont.)

Students are provided with a set of materials (e.g. newspaper articles, Federal Accident Report, e-mail exchanges, description and performance characteristics of AirSwift 235 and another model, etc.) and asked to prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors may have contributed to the accident and should be taken into account, and their overall recommendation about whether or not DynaTech should purchase the plane.

determinants of college learning dataset
Determinants of College Learning Dataset

Longitudinal Design

Fall 2005 and Spring 2007 (beginning of freshman and end of sophomore years)

Large Scale

24 diverse four-year institutions; 2,341 students

Breath of Information

Family background and high school information,

college experiences and contexts, college transcripts

Collegiate Learning Assessment (CLA)

research questions
Research Questions

What individual, social and institutional factors are associated with learning in higher education?

How do disadvantaged groups of students fare in college in terms of measured learning?

To what extent do individual, social and institutional factors account for variation across disadvantaged groups?

overview of the conceptual model employed in the study
Overview of the Conceptual Model Employed in the Study

Measures of Disadvantage:

Race/Ethnicity Parental Education Parental Occupation

Racially Segregated High School (70+ % minority) Non-English Language

Control Variables:

2005 Test Score Gender Two Parent Household

Sibling Number Urbanicity Geographic Region

2007 Test Score

High School Academic Preparation:

GPANumber of AP Courses Taken

College Experiences:

Hours Spent Studying Alone Hours Spent Studying with Peers

Hours Spent in a Fraternity/Sorority Hours Worked On/Off Campus

Faculty Expectations Field of Study

College Fixed Effects

analysis part i
Analysis - Part I

Individual, Social and Institutional Factors Associated with Learning as Measured by Improvement in CLA Performance

high school preparation
High School Preparation

Figure 1. Predicted 2007 Test Score by Number of High School AP Courses

college engagement and learning
College Engagement and Learning

Figure 2. Predicted 2007 Score by College Engagement and Involvement Measures

college employment and learning
College Employment and Learning

Figure 3. Predicted 2007 Test Score by Employment Measures

faculty expectations and learning
Faculty Expectations and Learning

Figure 4. Predicted 2007 Test Score by Level of Faculty Expectations

fields of study and learning
Fields of Study and Learning

Figure 5. Predicted 2007 Test Score by College Major

analysis part ii
Analysis - Part II

Social Disadvantaged Group Differences in Learning as Measured by Improvement in CLA Performance

cla performance by race
CLA Performance by Race

Figure 6. 2005 and 2007 Test Scores by Race

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

cla performance by parental education
CLA Performance by Parental Education

Figure 7. 2005 and 2007 Test Scores by Parental Education

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

cla performance by high school student composition and home language
CLA Performance by High School Student Composition and Home Language

Figure 8. 2005 and 2007 Test Scores by Level of High School Student Composition and Home Language

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

analysis part iii
Analysis - Part III

Accounting for Variation in CLA Performance by Social Disadvantaged Groups

accounting for group differences h s college experiences and institutional differences
Accounting for Group Differences: H.S.-College Experiences and Institutional Differences

Figure 11. Test score gaps in baseline and full models with college institutional fixed effects.

Note: Baseline regression model predicts the 2007 score, controlling for the 2005 score and a range of background characteristics. Full model also includes measures of high school academic preparation and college experiences. Non-significant differences are shaded.

conclusions and implications
Conclusions and Implications

Policy makers need to focus attention on improving individual student learning in higher education, not just access and retention.

Practitioners need to recognize the extent to which both student experiences as well as institutional differences are associated with variation in learning.

Additional systematic longitudinal research is necessary to improve understanding of these processes.

Measurement of learning across fields and institutions is possible with instruments such as the CLA.