Call for tenders oecd feasibility study for an ahelo
This presentation is the property of its rightful owner.
Sponsored Links
1 / 49

Call for Tenders: OECD Feasibility Study for an AHELO PowerPoint PPT Presentation


  • 52 Views
  • Uploaded on
  • Presentation posted in: General

Call for Tenders: OECD Feasibility Study for an AHELO. Assessment of Higher Education Learning Outcomes Information for bidders 9 July, 2009. Presentation. Part 1 – AHELO feasibility study Context Project overview Four strands of work and 10 participants

Download Presentation

Call for Tenders: OECD Feasibility Study for an AHELO

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Call for tenders oecd feasibility study for an ahelo

Call for Tenders:OECD Feasibility Study for an AHELO

Assessment of Higher Education Learning Outcomes

Information for bidders

9 July, 2009


Presentation

Presentation

  • Part 1 – AHELO feasibility study

  • Context

  • Project overview

  • Four strands of work and 10 participants

  • Expected outcomes and implications for the work

  • Governance structure

  • Part 2 – Terms of reference

  • Rationale for modular approach

  • Indicative timelines

  • Statement of work

  • Initial work provided for information

  • Part 3 – Submitting a proposal

  • Composition of bidders’ proposals

  • Evaluation process and criteria

  • Practical aspects, hearings and commercial negotiations


Part 1 ahelo feasibility study

Part 1 – AHELO feasibility study

  • Context

  • Project overview

  • Four strands of work and 10 participants

  • Expected outcomes of the feasibility study and implications for your work

  • Governance structure


Context why undertake the study

Context: why undertake the study?

Peer pressure and public accountability now more powerful than legislation and regulation

International comparisons now inevitable in this previously

national-only domain

Judgements about higher education outcomes will continue to be made on the basis of rankings derived from inputs or research-driven outputs

Action is challenging ...

But the cost of inaction is high too!

AHELO is not about ranking nor standardization, it is about

evidence for policy and practice


Moving from quantity to quality

Moving from Quantity to Quality...

Consensus on need to tackle quality challenge

2006: Tokyo Ministerial Meeting

But how? Information gap on LO…

2007: Experts meetings to explore the scope for an AHELO

Carry out a feasibility study to provide a proof of concept

Conclusion

  • From decision to action

  • 2008: Seoul Informal Ministerial Meeting

    • Launch of AHELO feasibility study, provided it takes into account institutional diversity

    • Recruitment of team and participating countries

  • 2009: Initial work and Call for Tenders


Project overview

Project overview…

What is AHELO?

  • A ground-breaking initiative to assess HE learning outcomes on an international scale, by creating measures that would be valid:

    • For all cultures and languages

    • And also for the diversity of HE institutions (not just universities)

Why is AHELO important?

  • Employs a wide range of measures

  • Provides a more balanced assessment of HE quality

  • No sacrifice of HEIs’ missions or autonomy in their subsequent efforts to improve performance

Yet, a debated initiative

  • Raises fears within some circles  Implications for how we run the project

    • Involving stakeholders in the process

    • Ensuring that the results will be un-attackable scientifically

    • Keeping budgets under control


The feasibility study at a glance

The feasibility study at a glance

To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible.

Goal?

Not a pilot, but rather a research approach to provide a proof of concept and proof of practicality.

What?

The outcomes will be used to assist countries to decide on the next steps.

Why?

The testing window is from August 2010 to April 2011

(with final results available by end 2011)

When?

Data will be collected from a targeted population of students who are near, but before, the end of their first 3-4 year degree.

Who?

OECD’s role is to establish broad frameworks that guide international expert committees and contractors charged with instrument development in the assessment areas.

How?


Multi dimensional def of quality

Multi-dimensional def° of quality

Addressing the needs of various users and uses

  • “Bottom line” of performance

  • “Value-added” to assess the quality of services

  • Contextual data to reveal best practices and problems, and to identify teaching and learning practices leading to greater outcomes

Both in discipline-related competencies …

  • Easily interpretable in the context of departments and faculties ...

  • But require highly differentiated instruments

And in generic skills

  • Less dependent on occupational and cultural contexts, applicable across HEIs …

  • But reflect cumulative learning outcomes and less relevant to the subject-matter competencies that are familiar to HEIs, departments or faculties


Implications of a feasibility study

Implications of a feasibility study…

… Limited timeframe compared to a fully-fledged study

  • This prevents us from developing a “perfect” instrument… but by using existing tools or instruments whose development is already underway, we can properly provide sufficient proof of concept

  •  Focus on cross-cultural appropriateness

… FEASIBILITY study, not a pilot study

  • This provides flexibility in exploring several directions/methodologies,

  • and allows some degree of risk-taking (e.g. unexplored areas)

  •  Request by countries to explore different methodologies/instruments

… Does not preclude the existence of a full-fledged study in the future

  • This allows us to be open-minded about the outcomes,

  • and about what its instruments might look like.

  •  It is expected that further work would be required for a main study.


Ahelo 4 strands of work

AHELO: 4 strands of work

Discipline strand

in Engineering

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data

Focus on ‘above content’ skills:

students’ ability to reflect, and to apply their knowledge and experience to novel and real world tasks and challenges

Discipline strand

in Economics

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data


Ahelo 4 strands of work1

AHELO: 4 strands of work

Generic skills strand

Discipline strand

in Engineering

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data

  • International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts.

    • + contextual data

  • With each assessment,

  • a collection of

  • contextual information:

    • to look beyond student performance: (e.g. institutional missions, selectivity, student characteristics and exposure to “good practices”, satisfaction).

    • to make AHELO an effective tool to reveal best practices and to identify shared problems.

Discipline strand

in Economics

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data


Ahelo 4 strands of work2

AHELO: 4 strands of work

Generic skills strand

Discipline strand

in Engineering

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data

  • International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts.

    • + contextual data

Research-based “Value-added” or “Learning gain” measurement strand

Discipline strand

in Economics

Several perspectives to explore the issue of value-added (conceptually, psychometrics), building on recent OECD work at school level.

  • Initial work on defining expected learning outcomes

  • through ‘Tuning’ approach.

  • + contextual data


Ahelo tests of instruments

3 assessment instruments

AHELO tests of instruments

Assessment generic skills

Assessment discipline-specific skills in engineering

Assessment discipline-specific skills in economics

Finland, Korea,

Mexico, Norway

Australia, Japan, Sweden

Belgium (Fl.), Italy, Mexico, Netherlands

  • Contextual instruments (4 surveys)

Contextual indicators and indirect proxies of quality

3 groups of countries


Participating countries

Participating countries

  • GenericSkills Strand

    • Finland

    • Korea

    • Mexico

    • Norway

  • Economics Strand

    • Belgium (FlemishCommunity)

    • Italy

    • Mexico

    • Netherlands

  • Engineering Strand

    • Australia

    • Japan

    • Sweden

  • Further countries?

    • Possible candidates: Canada (province level), Russia, Slovenia, South Africa, Spain, Turkey, US (state level)

    • Needs to betakenintoaccount in costing (marginal cost)


  • Expected outcomes of the feasibility study and implications for your work

    Expected Outcomes of the feasibility study and implications for your work

    Scientific and practical feasibility

    Goal?

    Focus of final report on feasibility aspects: cross-cultural validity, cultural biases, reliability issues etc.

    No publication of intal comparisons and no league tables

    Feasibility of implementation

    Focus on how to ensure student and faculty participation (or correct response biases)

    Need for some feedback (anonymous)

    Political feasibility

    Analyses to demonstrate the potential analytical value of AHELO for institutional improvement

    Not looking for a Rolls Royce…


    Remarks on data collection

    Remarks on data collection

    • No comparative data at the national level

    • Institutions/departments are the units of analysis, hence measures and reporting at HEI/dept level

    • Feedback to HEIs: performance profiles and contextual data, with their own results and those of other HEIs (anonymously)

    • Pragmatic and cost-effective mode of delivery


    Current status

    Current status

    • Institutional framework in place

    • Recruitment of partic. countries

      • Allocated to various strands of work to ensure geog. language & cultural diversity

  • Communication & dissemination

    • Web page, conferences, brochure

    • Stakeholders’ Consultative Group

  • Call for Tenders launched

  • Substantive work

    • Contextual dimension analyticalframework

    • Tuning-AHELO project to defineexpected LO in economics & engineering  reports

    • Project tentative timeline

    • Planning international test of CLA

    • Research-based value-addedstrand

  • Meeting of AHELO GNE: 3-4 Sept

  • Fundraisingcampaignunderway


  • Call for tenders oecd feasibility study for an ahelo

    Governance Structure

    EDPC

    PWB Objectives

    IMHE

    GB

    Contractors

    Generic Skills Strand Instrument

    Economics Strand Instrument

    Engineering Strand Instrument

    Contextual Dimension Instruments

    Project Management, Survey Operations and Analyses

    Regular reports

    Policy priorities

    Contracts and monitoring

    +

    Interface with GNE

    AHELO GNE

    Participating Countries

    NPMs

    Guidance on directions, priorities, design and conduct of FS

    OECD

    Secretariat

    Input, advice, dialogue

    Ad Hoc Experts Groups

    Experts Groups for initial work

    Technical Review Panel

    Technical Advisory Group

    Stakeholders’ Consultative Group

    Advice

    Interactiontowards implementation


    Part 2 terms of reference

    Part 2 – Terms of reference

    • Rationale for modular approach

    • Indicative timelines

    • Statement of work

    • Initial work provided for information


    Rationale for modular approach with a coordination module

    Rationale for Modular approach with a ‘coordination’ Module

    • Foster coherence and synergies across the different strands (3 strands but a common goal)

    • Benefits of more specialised expertise (few existing instruments to assess LO in HE)

    • Benefits of pooling expertise of different contractors towards a common goals (e.g. analysis)

    • Economies of scale (e.g. common sampling guidelines for all strands)

    • Streamline communications across strands

    • Incentive for cooperation between contractors

    • Benefits of enhanced competition at Module level


    Call for tenders oecd feasibility study for an ahelo

    AHELO feasibility study Modules

    Assessment instruments

    Generic Skills

    (Module A)

    Economics

    (Module B)

    Engineering

    (Module C)

    Methodological/Instrument development

    Contextual dimension surveys

    student / faculty /

    programme leadership / institutional leadership

    (Module D)

    Optional implementation of the

    Generic Skills instrument

    Project management, survey operations and analyses of results

    (Module E)

    Field implementation and analysis


    Section 3 statement of work

    Section 3: Statement of work

    Module Bdevelopment of an instrument for the Economics Strand

    Module C development of an instrument for the Engineering Strand

    Module D development of the Contextual Dimension surveys

    Module E overall project management, survey operations, and analysis of results


    Indicative timeline

    Indicative timeline

    • Constraint of results due end-2011

    • Constraint of academic years in different countries (testing windows)

    • Some degree of flexibility between milestones


    Call for tenders oecd feasibility study for an ahelo

    Planning

    Administering

    • Initial Work

    • CD

    • Learning Outcomes

    EngS + CD

    Australia

    Japan

    Sweden

    June ‘09

    Aug ‘10

    to

    Mar‘11

    GenS + CD

    Korea, Finland

    Norway

    Mexico

    Contract

    GenS

    July

    ‘09

    Nov ‘10

    to

    Apr ‘11

    EcoS + CD

    Netherlands

    Belgium (Fl.)

    Italy

    Mexico

    Contracts

    EcoS, EngS,

    CD, PMSO

    Feb ‘11

    to

    Apr ‘11

    Oct ‘09

    Nov ‘08

    Apr ‘11


    Call for tenders oecd feasibility study for an ahelo

    Planning

    Developing

    Administering

    • Initial Work

    • CD

    • Learning Outcomes

    Development of instruments and procedures

    EngS + CD

    Australia

    Japan

    Sweden

    June ‘09

    Aug ‘10

    to

    Mar‘11

    Translation

    and

    adaptation

    GenS + CD

    Korea, Finland

    Norway

    Mexico

    Contract

    GenS

    July

    ‘09

    Layout, printing and delivery

    Nov ‘10

    to

    Apr ‘11

    EcoS + CD

    Netherlands

    Belgium (Fl.)

    Italy

    Mexico

    NPM and IC training

    Contracts

    EcoS, EngS,

    CD, PMSO

    Feb ‘11

    to

    Apr ‘11

    Oct ‘09

    Coordination

    Nov ‘08

    Apr ‘11


    Call for tenders oecd feasibility study for an ahelo

    Administering

    Analysing

    Reporting

    Scoring

    and

    coding

    Technical report

    EngS

    EngS + CD

    Australia

    Japan

    Sweden

    Technical report

    PMSO

    Technical report

    VamS

    Technical report

    EcoS

    Technical report

    CD

    Technical report

    GenS

    Aug ‘10

    to

    Mar‘11

    Data cleaning

    and

    processing

    GenS + CD

    Korea, Finland

    Norway

    Mexico

    Final report

    Nov ‘10

    to

    Apr ‘11

    Scaling

    and

    analysis

    EcoS + CD

    Netherlands

    Belgium (Fl.)

    Italy

    Mexico

    Feb ‘11

    to

    Apr ‘11

    Coordination

    May ‘11

    Dec ‘11


    Call for tenders oecd feasibility study for an ahelo

    Reporting

    Disseminating

    Deciding

    Final confererencewith experts and stakeholders

    Final report

    The science of the assessment

    EDPC

    GNE

    IMHE GB

    The practicability of implementation

    Dec ‘11

    Nov‘11


    Statement of work

    Statement of work

    • Relationship between the contractors

    • General principles

    • Language to be used in meetings and documents

    • Development work undertaken to date


    Main tasks common to modules b c and d

    Main tasks common to Modules B, C and D

    • Develop frameworks: instruments/surveys;

    • Develop and prepare instruments/surveys;

    • Contribute to the documentation of instruments/surveys and technical reports;

    • Work with the contractor for Module E on test delivery and analysis of results; and

    • Work closely with the contractor(s) undertaking other Modules.


    Deliverables common to modules b c and d

    Deliverables common to Modules B, C and D

    • Framework s and specifications;

    • Instruments/surveys;

    • Reports mapping items to framework;

    • Scoring/coding guides; and

    • Contributions to the analysis plan and to the technical and final reports.


    Specific to modules b and c development of assessment instruments

    Specific to Modules B and CDevelopment of assessment instruments

    Assessment instruments

    • For two discipline-related competencies:

      Economics + Engineering

    • Aiming at ‘above content’ skills:

      Knowledge and skills essential for future life


    Specific to module d development of the survey instruments for the contextual dimension

    Specific to Module DDevelopment of the survey instruments for the Contextual Dimension

    • A student survey instrument (15 minutes);

    • A faculty survey instrument (15 minutes);

    • An institutional and a programme leadership surveys.

    Surveys instruments:


    Module e project management and survey operations

    Module EProject management and survey operations

    • Develop an overall assessment design:

      • meeting the aims of the feasibility study;

      • coordination across the various strands of work;

      • analysis plan to assess the cross-national and cross-cultural validity of the instruments.

    • Define the student target population;

    • Establish and manage two groups:

      • NPM + TAG

    • Develop package for test delivery and support tools accordingly;

    Main tasks:


    Module e project management and survey operations1

    Module EProject management and survey operations

    • Ensure the quality of translation/adaptation;

    • Establish survey procedures and operations;

    • Perform data verification and establish quality-control mechanisms;

    • Provide training and guidelines to NPMs and ICs;

    • Coordinate the implementation of the feasibility study;

    • Coordinate scaling and data analyses; and

    • Provide technical and analytical support for the final report.

    Main tasks (continued):


    Module e project management and survey operations2

    Module EProject management and survey operations

    • The overall assessment design with analyses to assess the cross-national and cross-cultural validity of the various instruments;

    • The analysis plan, including

      • specification of the research questions posed by the various dimensions;

      • proposed analyses; and

      • criteria to assess success in these various dimensions;

    • The student sampling plans including the sampling manual for NPMs/ICs;

    Deliverables:


    Module e project management and survey operations3

    Module EProject management and survey operations

    Deliverables(continued):

    • The translation/adaptation guidelines and report on the translation/adaptation issues;

    • The documentation on survey procedures and quality assurance procedures:

      • Test administration procedures;

      • Technical standards;

      • Coding guides; and

      • Data management manual.

    • Cleaned databases with the associated documented data product; and

    • Analyses, tables and technical documentation for the final report.


    Initial work provided for info

    Initial work provided for info°

    • Available from www.oecd.org/edu/ahelo/callfortenders

    • Dumais (2009), Higher education institutions, sampling issues preparing the AHELO feasibility study, OECD

    • Ewellet al.(2008), Report on the 1st meeting of experts on the AHELO contextual dimension, OECD

    • Ewell et al. (2009), Analytical framework for the contextual dimension of the AHELO feasibility study, OECD

    • Tuning Association (2009a), A Tuning-AHELO conceptual framework of expected/desired learning outcomes in the Science of Economics, OECD

    • Tuning Association (2009b), A Tuning-AHELO conceptual framework of expected/desired learning outcomes in Engineering, OECD


    Part 3 submitting a proposal

    Part 3 – Submitting a proposal

    • Composition of bidders’ proposals

    • Evaluation process and criteria

    • Practical aspects, hearings and commercial negotiations


    Composition of bidders proposals

    Composition of bidders’ proposals

    • 1. Budget/quotes

    • Disaggregated by Module

    • Including usage of staff time by task and year

    • Separate cost proposals for paper-and-pencil and e-delivery

    • Separate cost proposals for optional implementation of CLA

    • Including marginal cost of adding a participating country


    Composition of bidders proposals1

    Composition of bidders’ proposals

    • 2. Approach to the implementation of the tasks

    • Bidder's understanding of the project

    • Proposed approach to the implementation of the project tasks

    • Bidder‘s capacity to effectively undertake the project tasks (technical perspective, and high quality and timely delivery)


    Composition of bidders proposals2

    Composition of bidders’ proposals

    • 3. Organizational and management capacity

    • Bidder’s ability to develop collaborative working relationship with all actors/contractors and build consensus

    • Management structure (if sub-contractors), financial controls and quality assurance controls

    • Qualifications and experience of project staff

    • Organization’s past experience in large-scale assessment, surveys and international studies


    Evaluation process

    Evaluation process

    First by a Technical Review Panel (TRP)

    • 4 assessment experts and 2 disciplinary experts (prof. engineering, economist)

    • Signed agreement stating absence of a conflict of interest and commitment not to work for any of the bidders for the duration of the FS

    • Shortlist proposals and recommendation to AHELO GNE

    Then by the AHELO Group of National Experts

    • Shortlisted bidders will be invited to present their proposal to the AHELO GNE at its next meeting

    • (3-4 September, Paris)

    • Decision by AHELO GNE on the basis of TRP recommendation, outcomes of commercial negotiations, and presentations


    Technical review panel trp

    Technical Review Panel (TRP)

    Step 1 – Quality

    • Technical review of individual proposals and their scoring against 4 pre-defined quality criteria

    • Comparative evaluation of acceptable proposals on the basis of quality

    • Shortlist of acceptable proposals from quality perspective

    Step 2 – Value for money

    • Comparison of acceptable proposals on the basis of cost

    • Bonus points granted to those with best value for money

    • Overall comparative evaluation of trade-offs and recommendation to GNE

    • Selection of proposals to be presented on 3-4 Sept

    • Report to GNE


    Call for tenders oecd feasibility study for an ahelo

    TechnicalReviewProcess

    Final score

    Strengths and weaknesses

    Reading proposal

    Scoring

    Strengths and weaknesses

    Step 1 - Quality

    List of proposals meeting 80%

    Bonus points for value for money

    Module X

    List of selected proposals for Module X

    with final scores

    Individually

    Repeat for eachproposal and Module

    In group

    Step 2 – Value for money


    Evaluation criteria for quality

    Evaluation criteria for quality

    Technical quality

    (40 points)

    Relevant experience of organisation

    and its staff

    (25 points)

    Organisational and management capabilities

    (20 points)

    Innovation and efficiency gains

    (15 points)


    Practical aspects

    Practical aspects

    • To be sent by paper only!

    • 10 copies including one printable copy

    • Sealed envelopes with special note

    • See Article 3.1 for addressees

    • Deadline

    • Deadline for receipt: 5 August 2009 10am (Paris Time)

    • Contents

    • Approved and signed Tender

    • See Article 3.2 for supporting documents


    Hearings and commercial negotiations

    Hearings and commercial negotiations

    • Bidders whose proposals will be selected by the Technical Review Panel…

    • Will be offered a chance to propose a revised commercial offer (late August) following discussions with the Central Purchasing Group

    • Will be given an opportunity to present their proposals to the AHELO GNE (3-4 September)


    Decision making process

    Decision-makingprocess

    • Concluding report from Technical Review Panel

    •  Best proposals presented to AHELO GNE meeting

    • Commercial negotiations with pre-selected bidders

    • Decision by the AHELO GNE on the basis of proposals, TRP report and recommendation, and outcomes of commercial negotiations

    •  Recommendation of contractor(s) to OECD Procurement Board

    • Preparation, negotiation and approval of contractswith OECD Central Purchasing Group and LegalAffairsDirectorate

    • Procurement Board  Recommendation to OECD SG

    • Contractsigned by OECD SG


    Call for tenders oecd feasibility study for an ahelo

    • Questions


  • Login