Charlie lenth clenth@sheeo org
1 / 12

Charlie Lenth clenth@sheeo - PowerPoint PPT Presentation

  • Uploaded on

The Assessment of Higher Education Learning Outcomes (AHELO) OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations and Institutions SHEEO Higher Education Policy Conference August, 2012. Charlie Lenth

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Charlie Lenth clenth@sheeo' - oma

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Charlie lenth clenth@sheeo org

The Assessment of Higher Education Learning Outcomes (AHELO)OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations and InstitutionsSHEEO Higher Education Policy ConferenceAugust, 2012

Charlie Lenth

Ahelo context and development
AHELO Context and Development

  • OECD’s education studies, surveys and analyses

  • Near universal recognition of changing education needs and objectives, particularly at higher levels

  • 2006 Council of Education Ministers’ request to plan postsecondary assessment

  • OECD commissioned papers and convened groups of experts to develop AHELO design

Basic project objectives and design
Basic Project Objectives and Design

  • Feasibility study—to determine scientific and practical feasibility of assessing college learning across nations

  • Assess learning across and within the disciplines or program areas at degree completion

  • Minimum five diverse nations with non-scientific (convenience) sample of 10 institutions each

  • Small, random sample of graduating students with institutions as primary unit of analysis

  • Student, faculty, institutional contextual data to allow meaningful interpretation and analysis

U s involvement in ahelo
U.S. Involvement in AHELO

  • Considerable U.S. role in research design and development (CAE, Peter Ewell, ETS)

  • Invitation from OECD to SHEEO, and then to SHEEO members

  • U.S. participation in Generic Skills Strand since early 2010 and through many months of delays and uncertain funding (and several trips to Paris!)

U s financial support for ahelo participation
U.S. Financial Support for AHELO Participation

  • To OECD for assessment development: Lumina Foundation (early and largest foundation donor), SpencerFoundation, Teagle Foundation

  • For U.S. participation from Hewlett Foundation and U.S. Department of Education:

    • Payment of national participation fees

    • Discretionary grant by U.S. Secretary of Education in September 2011

    • In-kind contributions by institutions, states, NCHEMS and SHEEO

Country participation in ahelo
Country Participation in AHELO

Generic Skills Economics Engineering

Columbia Belgium Australia

Egypt Egypt Canada

Finland Italy Columbia

Korea Mexico Egypt

Kuwait Netherlands Japan

Mexico Russia Mexico

Norway Slovakia Russia

Slovakia Slovakia

United States

Total countries: 17 plus several EU country observers

Overview u s participants roles
Overview--U.S. Participants’ Roles

  • National level

    • US Dept of Education—member of OECD Education Governing Board

    • SHEEO—National Project Manager (NPM) and representative on project advisory board--Group of National Experts (GNE)

  • State level

    • SHEEO agency provides project leadership, coordination, and oversight in Connecticut, Missouri and Pennsylvania

  • Institutional level

    • AHELO Institutional Coordinator and “team”

    • IR office prepares student/faculty files and institutional survey

    • Test Administration—recruitment, scheduling, set up, test monitoring

    • Academic leadership—faculty involvement, organizational support

    • President, media relations, governing board, others

    • Anticipated analysis and use of returned assessment data

U s higher education institutions participating in ahelo feasibility study
U.S. Higher Education Institutions Participating in AHELO Feasibility Study

  • Connecticut Board of Regents for Higher Education

    • Southern Connecticut State University (public regional university; 9,000 undergraduates)

  • Missouri Department of Higher Education

    • Central Methodist University (private university; 3,500 students)

    • Missouri State University (public institution; 16,000 undergraduates)

    • Truman State University (public institution; 6,000 undergraduates)

    • University of Central Missouri (public regional; 12,000 students)

    • Webster University (private, not-for-profit; 4,000 undergraduates)

  • Pennsylvania State System of Higher Education (PASSHE)

    • Cheyney University of Pennsylvania (public historically black university; 1,300 undergraduates)

    • Clarion University (public institution; 5,100 undergraduates)

    • Edinboro University (public institution; 6,600 undergraduates)

    • Lock Haven University (public institution; 5,000 undergraduates)

    • Millersville University (public institution; 7,200 undergraduates)

Institutional sampling plan http www sheeo org misc ahelo 27samplingmanual pdf
Institutional Feasibility StudySampling Plan

  • AHELO Sampling Manual provides international guidelines for both student and faculty samples

  • Basic components:

    • Institutions provide census of “in scope” students and faculty to NPM (SPSS.sav or Excel .xls/.xlsx preferred)

    • NPM will draw equal probability samples using prescribed routines and submit for ACER review (generally 200 students and 40 faculty)

    • Flexibility for stratified or “augmented” sample design if justified and documented by institution (e.g., strata for major program areas, distinguishing student characteristics, special accommodations, etc.)

    • Review and approval by ACER Sampling Team

Generic skills assessment framework design
Generic Skills Feasibility StudyAssessment Framework/Design

  • Two rotating performance tasks adapted from Collegiate Learning Assessment (CLA) – 90 minutes

  • Selected response items from tested Australian item bank – 30 minutes

  • Brief student experience survey

  • Faculty/institutional web-based questionnaires

  • All computer-administered on secure international testing sites

Fieldwork analysis and reporting
Fieldwork, Analysis and Reporting Feasibility Study

  • Test administration by U.S. institutions using secure international testing websites

  • Lead scorer trained on international scoring rubrics recruited and managed five U.S scorers

  • Scored assessment results analyzed by ACER and reported to OECD; later to institutions

  • Preparation, review and release of Feasibility Study findings and recommendations

  • Final project conference March 2013

Context and consequences
Context and Consequences Feasibility Study

  • AHELO is part of a larger postsecondary education agenda with implications for students, institutions, nations and the global environment.

  • Much has already been learned about both scientific and practical feasibility for assessment within an international setting.

  • OECD and other major organizations will shape future decisions and financial commitments based in large part on this study.