The convergence of university rankings and system benchmarking
This presentation is the property of its rightful owner.
Sponsored Links
1 / 33

The Convergence of University Rankings and System Benchmarking PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on
  • Presentation posted in: General

The Convergence of University Rankings and System Benchmarking. An Apparent Paradox of “ Rankology ”. Questions. Two approaches : University Rankings System Benchmarking Are they: Complementary ? Competing? Consistent?. Outline. (1) Background: from ranking to benchmarking

Download Presentation

The Convergence of University Rankings and System Benchmarking

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The convergence of university rankings and system benchmarking

The Convergence of University Rankings and System Benchmarking

An Apparent Paradox

of “Rankology”


Questions

Questions

Two approaches:

University Rankings

System Benchmarking

Are they:

Complementary?

Competing?

Consistent?

IREG - Warsaw, 16-17 May 2013


Outline

Outline

(1) Background: from ranking to benchmarking

(2) Method of investigation

  • Results

    (4) Interpretation and conclusion

IREG - Warsaw, 16-17 May 2013


1 university rankings

(1) University Rankings

IREG - Warsaw, 16-17 May 2013


U rankings a polarizing exercise

U Rankings: a Polarizing Exercise

U Rankings:

hated/loved,

criticized/commended,

threatening/stimulating

but

proliferating (“here to stay”)

Ph. Albatch’s advice [“Don’t take too much notice of rankings” (UWN, March 23, 2013)]: unlikely to be widely followed

More pitfalls discovered, uncovered, elucidated more attempts to improve methods

IREG - Warsaw, 16-17 May 2013


U rankings the disease

U Rankings: the Disease

Methodological caveats

Biases: Research, English, STEM

Composite indicators: Weighting=> Elitism

Subjective (reputation) /non transparent

Dangerous use (“misuses”, “abuses”)

Universities:(1) Focus on competition with others instead of own improvement / Affect strategic planning

(2) Focus on biased criteria (research)

Policy makers: Focus on a few WCUs

instead of whole system

Students:Impact on university selection

Overall: Impact on financing

Commercialization(crowded) market

IREG - Warsaw, 16-17 May 2013


From ranking to benchmarking

From Ranking to Benchmarking

“If Ranking is the Disease,

Is Benchmarking the Cure?”

(JamilSalmi, SunitaKosaraju. Evaluation in Higher Education, Vol. 5 no.1, June 2011)

“Rankings: Neither a Disease nor a Cure”

(Ph. Albatch, UWN, 2013)

IREG - Warsaw, 16-17 May 2013


2 system benchmarking

(2) System Benchmarking

Governance

Resources

TE SYSTEM

Access

Quality control

Private Providers

Equity

Economic, Social & Technological Environment

IREG - Warsaw, 16-17 May 2013


Benchmarking objective criteria

Benchmarking: Objective & Criteria

Objective: assess strength, health and performance of countries' tertiary education systems

Criteria: resources, inputs, governance, outputs and outcomes of the system (access, equity, quality, relevance)

IREG - Warsaw, 16-17 May 2013


Benchmarking main initiatives

Benchmarking: Main Initiatives

  • SABER: System Approach for Better Education Results (World Bank) Still under construction

  • U21(Universitas 21/ University of Melbourne)Most recent, comprehensive available case  See below

  • Benchmarking University Governance (World Bank – MENA): Hybrid

  • AHELO: Assessment of Higher Education Learning Outcomes (OECD)

     Still under experimentation

IREG - Warsaw, 16-17 May 2013


Hypothesis

Hypothesis

Benchmarking developed in reaction to Rankings

Objectives, level of observation and criteria of Benchmarking and Ranking are quite different

==Shouldn’t they yield different results?

IREG - Warsaw, 16-17 May 2013


Method 1

Method (1)

1/ Select 4 of the more popular university rankings:

ARWU, THE, QS, WEBOmetrics

2/ Pick the most recent system benchmarking: U21

3/ Compare their results

IREG - Warsaw, 16-17 May 2013


Method 2

Method (2)

Issue: How to compare U and Systems?

Solution:Translate U rankings into Country Rankings

Method: From: number of top universities

to: number of tertiary aged youths in one country potentially served by top universities in that country

(e.g. supply of top universities)

NB: no correlation between the 2 measures

IREG - Warsaw, 16-17 May 2013


Nb number of top 400 u and supply of top 400 u the rank

NB: Number of Top 400 U and Supply of Top 400 U (THE) : Rank)

IREG - Warsaw, 16-17 May 2013


Method 3

Method (3)

Quick look at the 4 leagues selected

The “sample”: Top 400 universities

IREG - Warsaw, 16-17 May 2013


The 34

The 34


Comparing the results of the 4 rankings 1

Comparing the results of the 4 Rankings (1)

Correlation between results of the 4 leagues:

(Number of top universities in each country)

IREG - Warsaw, 16-17 May 2013


Comparing the results of the 4 rankings 2

Comparing the results of the 4 Rankings (2)

Correlation between results of the 4 leagues:

(1) number of top universities in each country

IREG - Warsaw, 16-17 May 2013


Comparing the results of the 4 rankings 3

Comparing the results of the 4 Rankings (3)

Correlation between results of the 4 leagues:

(2) Supplyof top universities

IREG - Warsaw, 16-17 May 2013


Supply nbr of top u te aged population

Supply: Nbr of top U/ TE aged population

The first five countries

QSARWUTHEWEBO

1Finland16.16.911.59.2

2New Zealand14.54.814.52.4

3Switzerland13.411.813.411.8

4Ireland13.38.013.35.3

5Denmark11.59.211.59.2

The last five countries

QSARWUTHEWEBO

30Poland0.30.50.50.8

31Mexico0.10.10.10.1

32Brazil0.10.20.10.4

33China0.10.10.10.2

34India0.040.010.020.01


Benchmarking u 21 method 1

Benchmarking: “U 21”Method (1)

1/ A priori selection of 48 countries ( +2)

2/ Assessment of countries’ performance based on one overall indicator and 4 “measures”:

(1) Resources (2) Environment

(3)Connectivity (4)Output

IREG - Warsaw, 16-17 May 2013


Benchmarking method 2

Benchmarking: Method (2)

  • Resources (25%):

    5 indicators on expenditures

    (2) Environment (25%):

    2 indicators on gender balance,

    1 indicator on data quality,

    3 indicators on policy and regulatory environment,

    1 homegrown index on internal governance

IREG - Warsaw, 16-17 May 2013


Benchmarking method 3

Benchmarking: Method (3)

(3) Connectivity (10%):

2 indicators on degree of internationalization (students & research)

(4) Output (40%):

5 indicators on research,

1 indicator on Probability of a person to attend a top 500 university (*) based on ARWU…

1 indicator on enrollment

1 indicator on tertiary educated population

1 indicator on unemployment among tertiary educated population

IREG - Warsaw, 16-17 May 2013


Benchmarking links between the 5 measures

Benchmarking: Links between the 5 measures

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 1a

Comparing Results of Rankings and Benchmarking (1a)

Countries Overlap between UR and SB:

U21 & THE:37 common countries

U21 & QS:40 common countries

U21 & ARWU:37 common countries

U21 & WEBO:41 common countries

  • Essentially same pool of countries

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 1b

Comparing Results of Rankings and Benchmarking (1b)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 2

Comparing Results of Rankings and Benchmarking (2)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 3

Comparing Results of Rankings and Benchmarking (3)

U21 (Overall) and THE Rankings (R2= 0.74)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 4

Comparing Results of Rankings and Benchmarking (4)

U21 (Resources) & ARWU (Supply): R2 = 0.78

IREG - Warsaw, 16-17 May 2013


Conclusions interpretation

Conclusions /Interpretation

1/Hypothesis not confirmed:

a/same set of countries

b/similar results

2/Two types of explanations:

a/methodological

b/structural

IREG - Warsaw, 16-17 May 2013


Epilogue

Epilogue

  • System Benchmarking ends up ranking countries

  • Boundaries between UR and SB are blurred

  • SB suffers common symptoms with UR

  • Convergence of the two streams of “Rankology” not surprising

  • Benchmarking needs to expand its pool of countries to become more relevant

IREG - Warsaw, 16-17 May 2013


Take away

Take Away

IREG - Warsaw, 16-17 May 2013


Thank you

Thank You

SB

UR

IREG - Warsaw, 16-17 May 2013


  • Login