Assessment of Higher Education Learning Outcomes:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 16

Diane Lalancette Directorate for Education Organisation for Economic Cooperation and Development PowerPoint PPT Presentation


  • 48 Views
  • Uploaded on
  • Presentation posted in: General

Assessment of Higher Education Learning Outcomes: a ground-breaking initiative to assess quality in higher education on an international scale. Diane Lalancette Directorate for Education Organisation for Economic Cooperation and Development UNICA Rectors’ Seminar 2010 Tallinn 28 May 2010.

Download Presentation

Diane Lalancette Directorate for Education Organisation for Economic Cooperation and Development

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Diane lalancette directorate for education organisation for economic cooperation and development

Assessment of Higher Education Learning Outcomes:a ground-breaking initiative to assess quality in higher education on an international scale

Diane Lalancette

Directorate for Education

Organisation for Economic Cooperation and Development

UNICA Rectors’ Seminar 2010

Tallinn

28 May 2010


What we know about he quality

What we know about HE quality…

The massification of participation in higher education has meant much more heterogeneous abilities and expectations of students than in the past

  • Proxies of higher education quality exist, but none are perfect

    • Rankings often biased towards input factors and research

    • Subjectivity of reputation factor

    • Cultural sensitivity of satisfaction factor

    • Labour market outcomes sensitive to conjuncture and local economic conditions

So what?

  • Learning outcomes as a promising direction

  • Defining them (Tuning process in Bologna area)

  • Incorporating them in quality assurance processes

  • Measuring them → AHELO


Objectives

Objectives

What is AHELO?

  • A ground-breaking initiative to assess what students know and can do upon graduation

  • Creating measures that would be valid:

    • For all cultures and languages;

    • And also for the diversity of HE institutions.

Why undertake the study?

  • After decades of quantitative growth in HE, consensus on the need to ensure quality for all (Athens, 2006)… but information gap on learning outcomes

  • Carry out a feasibility study to provide a proof of concept (Tokyo, 2008)

Why is AHELO important?

  • Employs a wide range of measures

  • Provides faculties, students and government agencies with a more balanced assessment of HE quality – not just research-driven rankings!

  • No sacrifice of HEIs’ missions or autonomy in their subsequent efforts to improve performance.


Multi dimensional definition of quality

Multi-dimensional definition of quality

Addressing the needs of various users and uses

  • “Bottom line” of performance

  • “Value-added” to assess the quality of services

  • Contextual data to reveal best practices and problems, and to identify teaching and learning practices leading to greater outcomes

Both in discipline-related competencies …

  • Easily interpretable in the context of departments and faculties ...

  • But require highly differentiated instruments

And in generic skills

  • Less dependent on occupational and cultural contexts, applicable across HEIs …

  • But reflect cumulative learning outcomes and less relevant to the subject-matter competencies that are familiar to HEIs, departments or faculties


The feasibility study at a glance

The feasibility study at a glance

To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible.

Goal?

Not a pilot, but rather a research approach to provide a proof of concept and proof of practicality.

What?

The outcomes will be used to assist countries to decide on the next steps. No pre-determined conclusion.

Why?

Phase 1 cognitive labs: January 2010 to June 2011

Phase 2 implementation: late 2011-2012

When?

Target population of students who are near, but before, the end of their first 3-4 year degree.

Who?

OECD guiding international expert committees

Contractors in charge of developing the assessment instruments.

How?


Remarks on data collection

Remarks on data collection

  • No comparative data at the national level

  • Institutions/departments are the units of analysis, hence measures and reporting at HEI/dept level

  • Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)


Ahelo 4 strands of work

AHELO: 4 strands of work

Generic skills strand

Discipline strand

in Engineering

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts.

  • + contextual data

  • + contextual data

  • With each assessment,

  • a collection of

  • contextual information:

    • to look beyond student performance: (e.g. institutional missions, selectivity, student characteristics and exposure to “good practices”, satisfaction).

    • to make AHELO an effective tool to reveal best practices and to identify shared problems.

Discipline strand

in Economics

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data


Ahelo 4 strands of work1

AHELO: 4 strands of work

Generic skills strand

Discipline strand

in Engineering

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data

  • International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts.

    • + contextual data

Research-based “Value-added” or “Learning gain” measurement strand

Discipline strand

in Economics

Several perspectives to explore the issue of value-added (conceptually, psychometrics), building on recent OECD work at school level.

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data


Ahelo tests of instruments

3 assessment instruments

AHELO tests of instruments

Australia

Japan

Sweden

Generic skills

Engineering

Economics

Belgium (Fl.)

Italy

Mexico

Netherlands

Russian Federation

Colombia (tbc)

Finland

Kuwait

Korea

Mexico

Norway

USA

  • 2 contextual instruments

Contextual indicators and indirect proxies of quality

to analyse determinants of LO and their development

3 groups of countries


Work to be undertaken in 2 phases

Work to be undertaken in 2 phases

Generic Skills

Framework

(started)

Economics

Framework

(underway)

Engineering

Framework

(underway)

Frameworks

Phase 1 -Initial proof of concept

Instrument development & small-scale validation

Generic Skills

Instrument

(started)

Economics

Instrument

(underway)

Engineering

Instrument

(underway)

Contextual dimension surveys

Phase 2 -Scientific feasibility

& proof of practicality

Project management,

survey operations and

analyses of results

Implementation


Practical considerations

Practical considerations

  • Test of practicality of implementation: international standards for test administration and student participation rates within HEIs

  • Assessments possibly computer-delivered or web-based

  • Performance described through proficiency levels and “can-do” statements

  • Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)

  • Use existing instruments, respecting their integrity; orcombine item pools of existing instruments in ways that cover frameworks; ordevelop new test material


Imagine quality education in 2017

Imagine quality education in 2017

  • If AHELO were one of the QA instruments available to HEIs…

    • HEIscoulddecide to participate… or not

    • HEIscoulddecide to use it for internalimprovementonly, or makeresults public

  • What AHELO could do…

    • Benchmark HEIs against their peers to identify strengths and weaknesses

    • Assist HEIs’ with student recruitment and students’ choices

    • Limit academic drift by showcasing result’s within one’s mission

    • Enable researchers to investigate teaching and learning processes at depth: What works and under which contexts/conditions

    • Spur QA reflection

  • What AHELO would not do…

    • Rank countries and/or HEIs…

    • Provide a miracle solution to accountability requirements: learning outcomes are only one aspect of the mission of HEIs (along with equity, regional mission, etc) There is no single criterion in HE quality space

    • Replace QA


A study with great potential

A study with great potential…

… Diagnosis is the basis of any improvement

  • Better information on student learning outcomes is the first step to

  • improve teaching and learning for all:

  • Provide evidence for national and institutional policy and practice

  • Equip institutions with the method and tools to improve teaching

… Shaping the future of higher education to address key challenges

  • Equity

  • Build fairer higher education systems, promoting success for all

  • Responsiveness

  • Better connect higher education and society

  • Effectiveness

  • Help students make informed choices to ensure success for all

  • Impact

  • Foster international transparency and mobility

… Scope for a few more participants


Ahelo is managed by the oecd imhe programme institutional management in higher education

AHELO is managed by the OECD IMHE Programme Institutional Management in Higher Education


Thank you

Thank you!

Please send

questions or comments to

[email protected]

For more information, visit

www.oecd.org/edu/ahelo

Thank you


  • Login