The convergence of university rankings and system benchmarking
Download
1 / 33

The Convergence of University Rankings and System Benchmarking - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on
  • Presentation posted in: General

The Convergence of University Rankings and System Benchmarking. An Apparent Paradox of “ Rankology ”. Questions. Two approaches : University Rankings System Benchmarking Are they: Complementary ? Competing? Consistent?. Outline. (1) Background: from ranking to benchmarking

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

The Convergence of University Rankings and System Benchmarking

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The convergence of university rankings and system benchmarking
The Convergence of University Rankings and System Benchmarking

An Apparent Paradox

of “Rankology”


Questions
Questions

Two approaches:

University Rankings

System Benchmarking

Are they:

Complementary?

Competing?

Consistent?

IREG - Warsaw, 16-17 May 2013


Outline
Outline

(1) Background: from ranking to benchmarking

(2) Method of investigation

  • Results

    (4) Interpretation and conclusion

IREG - Warsaw, 16-17 May 2013


1 university rankings
(1) University Rankings

IREG - Warsaw, 16-17 May 2013


U rankings a polarizing exercise
U Rankings: a Polarizing Exercise

U Rankings:

hated/loved,

criticized/commended,

threatening/stimulating

but

proliferating (“here to stay”)

Ph. Albatch’s advice [“Don’t take too much notice of rankings” (UWN, March 23, 2013)]: unlikely to be widely followed

More pitfalls discovered, uncovered, elucidated more attempts to improve methods

IREG - Warsaw, 16-17 May 2013


U rankings the disease
U Rankings: the Disease

Methodological caveats

Biases: Research, English, STEM

Composite indicators: Weighting=> Elitism

Subjective (reputation) /non transparent

Dangerous use (“misuses”, “abuses”)

Universities:(1) Focus on competition with others instead of own improvement / Affect strategic planning

(2) Focus on biased criteria (research)

Policy makers: Focus on a few WCUs

instead of whole system

Students:Impact on university selection

Overall: Impact on financing

Commercialization(crowded) market

IREG - Warsaw, 16-17 May 2013


From ranking to benchmarking
From Ranking to Benchmarking

“If Ranking is the Disease,

Is Benchmarking the Cure?”

(JamilSalmi, SunitaKosaraju. Evaluation in Higher Education, Vol. 5 no.1, June 2011)

“Rankings: Neither a Disease nor a Cure”

(Ph. Albatch, UWN, 2013)

IREG - Warsaw, 16-17 May 2013


2 system benchmarking
(2) System Benchmarking

Governance

Resources

TE SYSTEM

Access

Quality control

Private Providers

Equity

Economic, Social & Technological Environment

IREG - Warsaw, 16-17 May 2013


Benchmarking objective criteria
Benchmarking: Objective & Criteria

Objective: assess strength, health and performance of countries' tertiary education systems

Criteria: resources, inputs, governance, outputs and outcomes of the system (access, equity, quality, relevance)

IREG - Warsaw, 16-17 May 2013


Benchmarking main initiatives
Benchmarking: Main Initiatives

  • SABER: System Approach for Better Education Results (World Bank) Still under construction

  • U21(Universitas 21/ University of Melbourne)Most recent, comprehensive available case  See below

  • Benchmarking University Governance (World Bank – MENA): Hybrid

  • AHELO: Assessment of Higher Education Learning Outcomes (OECD)

     Still under experimentation

IREG - Warsaw, 16-17 May 2013


Hypothesis
Hypothesis

Benchmarking developed in reaction to Rankings

Objectives, level of observation and criteria of Benchmarking and Ranking are quite different

==Shouldn’t they yield different results?

IREG - Warsaw, 16-17 May 2013


Method 1
Method (1)

1/ Select 4 of the more popular university rankings:

ARWU, THE, QS, WEBOmetrics

2/ Pick the most recent system benchmarking: U21

3/ Compare their results

IREG - Warsaw, 16-17 May 2013


Method 2
Method (2)

Issue: How to compare U and Systems?

Solution:Translate U rankings into Country Rankings

Method: From: number of top universities

to: number of tertiary aged youths in one country potentially served by top universities in that country

(e.g. supply of top universities)

NB: no correlation between the 2 measures

IREG - Warsaw, 16-17 May 2013


Nb number of top 400 u and supply of top 400 u the rank
NB: Number of Top 400 U and Supply of Top 400 U (THE) : Rank)

IREG - Warsaw, 16-17 May 2013


Method 3
Method (3)

Quick look at the 4 leagues selected

The “sample”: Top 400 universities

IREG - Warsaw, 16-17 May 2013



Comparing the results of the 4 rankings 1
Comparing the results of the 4 Rankings (1)

Correlation between results of the 4 leagues:

(Number of top universities in each country)

IREG - Warsaw, 16-17 May 2013


Comparing the results of the 4 rankings 2
Comparing the results of the 4 Rankings (2)

Correlation between results of the 4 leagues:

(1) number of top universities in each country

IREG - Warsaw, 16-17 May 2013


Comparing the results of the 4 rankings 3
Comparing the results of the 4 Rankings (3)

Correlation between results of the 4 leagues:

(2) Supplyof top universities

IREG - Warsaw, 16-17 May 2013


Supply nbr of top u te aged population
Supply: Nbr of top U/ TE aged population

The first five countries

QSARWUTHEWEBO

1Finland16.16.911.59.2

2New Zealand14.54.814.52.4

3Switzerland13.411.813.411.8

4Ireland13.38.013.35.3

5Denmark11.59.211.59.2

The last five countries

QSARWUTHEWEBO

30Poland0.30.50.50.8

31Mexico0.10.10.10.1

32Brazil0.10.20.10.4

33China0.10.10.10.2

34India0.040.010.020.01


Benchmarking u 21 method 1
Benchmarking: “U 21”Method (1)

1/ A priori selection of 48 countries ( +2)

2/ Assessment of countries’ performance based on one overall indicator and 4 “measures”:

(1) Resources (2) Environment

(3)Connectivity (4)Output

IREG - Warsaw, 16-17 May 2013


Benchmarking method 2
Benchmarking: Method (2)

  • Resources (25%):

    5 indicators on expenditures

    (2) Environment (25%):

    2 indicators on gender balance,

    1 indicator on data quality,

    3 indicators on policy and regulatory environment,

    1 homegrown index on internal governance

IREG - Warsaw, 16-17 May 2013


Benchmarking method 3
Benchmarking: Method (3)

(3) Connectivity (10%):

2 indicators on degree of internationalization (students & research)

(4) Output (40%):

5 indicators on research,

1 indicator on Probability of a person to attend a top 500 university (*) based on ARWU…

1 indicator on enrollment

1 indicator on tertiary educated population

1 indicator on unemployment among tertiary educated population

IREG - Warsaw, 16-17 May 2013


Benchmarking links between the 5 measures
Benchmarking: Links between the 5 measures

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 1a
Comparing Results of Rankings and Benchmarking (1a)

Countries Overlap between UR and SB:

U21 & THE:37 common countries

U21 & QS:40 common countries

U21 & ARWU:37 common countries

U21 & WEBO:41 common countries

  • Essentially same pool of countries

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 1b
Comparing Results of Rankings and Benchmarking (1b)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 2
Comparing Results of Rankings and Benchmarking (2)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 3
Comparing Results of Rankings and Benchmarking (3)

U21 (Overall) and THE Rankings (R2= 0.74)

IREG - Warsaw, 16-17 May 2013


Comparing results of rankings and benchmarking 4
Comparing Results of Rankings and Benchmarking (4)

U21 (Resources) & ARWU (Supply): R2 = 0.78

IREG - Warsaw, 16-17 May 2013


Conclusions interpretation
Conclusions /Interpretation

1/Hypothesis not confirmed:

a/same set of countries

b/similar results

2/Two types of explanations:

a/methodological

b/structural

IREG - Warsaw, 16-17 May 2013


Epilogue
Epilogue

  • System Benchmarking ends up ranking countries

  • Boundaries between UR and SB are blurred

  • SB suffers common symptoms with UR

  • Convergence of the two streams of “Rankology” not surprising

  • Benchmarking needs to expand its pool of countries to become more relevant

IREG - Warsaw, 16-17 May 2013


Take away
Take Away

IREG - Warsaw, 16-17 May 2013


Thank you
Thank You

SB

UR

IREG - Warsaw, 16-17 May 2013


ad
  • Login