Quality and Inequality in Academic Labor Markets

1. Quality and Inequality in Academic Labor Markets by James Moody The Ohio State University

2. Quality & Inequality in Academic Labor Markets • Introduction & Background • Academic Caste Systems • Suggestive Findings from a sociology market • A reasonable null? • Simulation Setup • Market Elements • An example run • Simulation Results • Market Clearing • Size & Quality • Position Stability • Academic Castes? • Tentative Conclusions • Some mechanisms • Potential implications • Future changes / extensions

3. Introduction & Background Academic Caste Systems • Merton (1942,1968) • Two key features that shape the academic market: • Universalistic criteria to evaluate quality • “Mathew effect:” the cumulative advantage of prestige • Burris (2004:239) states as fact that prestige is ascribed rather than achieved, arguing that • “Moreover, through a process of cumulative advantage, academic scientists and scholars who secure employment in the more prestigious departments gain differential access to resources and rewards that enhance their prospects. This cycle results in a stratified system of departments and universities, ranked in terms of prestige, that is highly resistant to change.” (p.239) • Burris attribute’s much of this stability to “Social Capital” in the PhD hiring market.

4. Introduction & Background Academic Caste Systems • Two types of evidence are used to demonstrate non-universalistic effects: • A less-than-perfect association between measures of faculty productivity and department prestige (Long, Hargens, Jacobs, Baldi, Burris) • Burris shows that between 30% and 50% of the variance in NRC rankings can be accounted for with standard productivity measures, leaving the remainder for non-meritocratic factors. • A strong correlation between simple number of faculty and prestige (r = 0.63 in sociology). • Probability / prestige of first job due to origin of PhD rather than publication record (but see Cognard-Black, 2004 and below).

5. Introduction & Background Academic Caste Systems Two types of evidence are used to demonstrate non-universalistic effects: 2. An extreme stability of department rankings over time Burris, ASR 2004 The correlation in NRC faculty quality scores in Sociology from 1982 to 1993 is 0.92

6. Introduction & Background Academic Caste Systems The combined effect becomes clear in the PhD exchange network: Hanneman (2001), overlapping PhD exchange networks, Sociology

7. Introduction & Background Academic Caste Systems The combined effect becomes clear in the PhD exchange network Hanneman (2001), overlapping PhD exchange networks, Sociology

8. I (N=52) II-a (N=4) II-b (N=15) II-c (N=22) II-d (N=81) III (N=384) Introduction & Background Academic Caste Systems Han, S-K. Social Networks 2003:251-280. Figure 1

9. Introduction & Background Academic Caste Systems The resulting status-based network has a strong correlation between centrality in the hiring network & quality ranking “Social Capital” = Bonacich Centrality on symmetric version of the PhD exchange Network

10. Introduction & Background Academic Caste Systems Can we square this stability & centrality with universalistic scientific norms? First, research on markets and cultural consumption suggests that quality is accurately perceived particularly when external measures show small differences (White 2002, J. Blau, Bourdieu). “Quality exists, whether it's defined or not. ” - Robert Pirsig (1972) That is, we know quality even if our systematic measures of quality are poor, which is reflected (in part) through market convergence on particular candidates (see below).

11. Introduction & Background Academic Caste Systems Can we square this stability & centrality with universalistic scientific norms? Second, most data on the market structure systematically selects on the dependent variable, as only those who are eventually hired are observed. This has the effect of: a) limiting variation on observed quality measures b) makes it impossible to disentangle PhD volume from placement patterns Recent dissertation work by Cognard-Black, for example, shows that the independent effect of PhD institution on placement is often lower than publication quality measures, once you expand the sample of PhDs beyond those hired to major research universities.

12. Introduction & Background Suggestive Findings from Sociology Further evidence: a sample based on all applicants for an open position • Data from the OSU jr. recruitment committee last year: • Systematically code productivity (a function of (1) number of publications, (2) weighted by impact factor of the outlet, (3) type of publication (book > article > chapter > review), and authorship order. • Followed all applicants through the process to see where people take jobs. • Data are limited to OSU applicants (but to an open position, and we have people from all ranks who take jobs at all ranks) and only have 1-side of the ‘matching’ process (i.e. we don’t know where people applied).

13. Introduction & Background Suggestive Findings from Sociology Further evidence: a sample based on all applicants for an open position

14. Top Producers Line Introduction & Background Suggestive Findings from Sociology Further evidence: a sample based on all applicants for an open position Regression line

15. Introduction & Background Academic Caste Systems Further evidence: a sample based on all applicants for an open position Job Placement Treat this distribution as a ranked outcome, and model by productivity & prestige Hired at Non-PhD Granting Institution No Job Bottom (51+) Dept 50 - 21 Dept 20 – 11 Dept Top 10 Dept Based on 116 new PhDs applying to the OSU open search in 2004

16. Introduction & Background Academic Caste Systems Further evidence: a sample based on all applicants for an open position Based on coefficients from an ordered logistic regression model for job placement rank, using 116 new PhDs applying to the OSU open search in 2004 (model also controls for minority status & gender)

17. Introduction & Background Identifying a Reasonable Null What should the PhD production system look like? In systems with open markets, merit-based hiring & rational actors: 1) How stable will quality rankings be? 2) Will size and quality be correlated? 3) Will network exchange centrality predict quality? Each has been used as evidence for non-meritocratic prestige systems, but we don’t know how the observed cases match the expected cases, because we have no reasonable null distribution. A key advantage of using a simulation is to identify a range of reasonable null distributions.

18. Introduction & Background Structure from Action A key question in sociology is where structure comes from. A long line of simulation studies have show how very simple individual rules can generate complex global patterns: Schelling on racial segregation Axlerod on systems of cooperation Epstein & Axtell's “Sugarscape” for inequality In all of these cases, we can often generate a macro-system with all of the relevant characteristics (spatial segregation, high gini coefficients) from very clear local behavior that is indifferent to the global features.

19. Introduction & Background Two Sided Matching Markets • A long-line of work (building on Roth), focuses on the incentive structure and performance of markets where two sets of actors rank each other. • Non-academic examples include: • Artists and galleries • Medical interns and hospitals • Rushees and Greek houses • Law graduates and Federal Clerkships • These markets are distinguished from commodity markets in that goods are not easily substitutable, there is usually a constrained time-frame for action in the market, and each “side” of the market plays an active role in completing the market transaction. • The market is characterized by two ‘dirty sorts’  where each side ranks the other to make an exchange.

20. Introduction & Background Two Sided Matching Markets • Two-sided matching markets are famous for their dramatic failures: • “Market unraveling” where timing is pushed ever earlier to jump the competition (rushing high-school students, appointing 1st year students to clerkships, etc.) • Exchanges that do not please all/most actors • Holding places / offers to “trade up” • “Opportunistic” contract arrangements • Many of these failures have two things in common: • They rob actors of information necessary to make good choices • They result in placements that do not maximize preferences

21. Introduction & Background Two Sided Matching Markets The Sociology market, for example, is clearly inching toward unraveling: Typical application dates are moving up, and variance is becoming smaller. Nov 1 Oct 22 Oct 23 Oct 15 Oct 7 Jul 19 Aug 3 Aug 18 Sep 18 Sep 17 Oct 2 Oct 17 Nov 1 Nov 16 Dec 1 Dec 16 Dec. 31

22. Introduction & Background Two Sided Matching Markets While economists have focused on identifying the conditions necessary to solve such inefficiencies, they have paid much less interest to how choice-relevant factors in these markets affect organizational structures and performance. By systematically varying the market features that shape the “dirty sorts” driving such markets, we can generate null hypotheses for questions about market prestige stability, exchange hierarchy and overall quality.

23. Simulation Setup Purpose & Questions • The purpose of this simulation is to examine the effect of market-relevant behavior under ideal-typical conditions. This involves simplifying the real world as much as possible, to isolate how particular factors affect outcomes of interest. • Key real-world properties of interest: • Stable prestige / quality rankings • Strong correlation between size and quality • Centralized hiring networks • Strong correlation between centrality, prestige, & size • Currently all actors in the simulation follow the same strategies, which vary across simulation runs. A future goal is to vary department strategies within runs to see what features lead to competitive advantage.

24. Simulation Setup Market elements • Actors • Departments: Collections of faculty who hire applicants & produce new students. (N=100). • Initial department size is drawn from a normal distribution with mean = 25, std=12, but I re-draw if size is less than 10, so the actual distribution is a truncated normal. • Applicants: Students from (other) departments who apply for jobs. • Departments seek to hire the best students, students want to work at the best departments. • Both actors are rational, honest, and risk-averse. But all actors have individual preferences / errors in vision.

25. Simulation Setup Market elements • Attributes • Quality. Each faculty member and student has an overall quality score. • Initial faculty quality is distributed as random normal(0,1). • Student quality is a (specifiable) random function of faculty quality. • Departments are rated based on the mean of faculty quality. • While each person has a set “real” quality score, actor choices are made based on an evaluation of quality that varies across actors. • This variation reflects jointly differences in preferences and ability to discern quality from production.

26. Simulation Setup Market elements • Action: Departments • Departments hire & produce students. • For each of 100 years: • Every department produces students (conditional on size). • A (random) subset of departments have job openings based on retirements in the prior year & current size relative to their resource-based target size. • Departments rank applicants by their evaluation of applicant quality, and make offers to their top choices. • If a department’s 1st choice goes elsewhere, they go to next for a specifiable number of rounds to a specifiable ‘depth’ into the pool. • Jobs can go unfilled, which means that departments can both grow and shrink

27. Simulation Setup Market elements Action: Departments The probability a job opening in any given year is a function of size & retirements (1-year replacement):

28. Simulation Setup Market elements Action: Departments Faculty size decreases through retirement

29. Simulation Setup Market elements • Action: Students / Applicants • Students rank departments that make them an offer by their evaluation of department quality, and take the best job they are offered. • If an applicant does not receive a job offer in a given year, they move out of the system • Lots of applicants don’t get jobs (at PhD granting universities…) • Applicants are not strategic: they do not hold a good offer while waiting for a better one (though this could be added)

30. Simulation Setup Variable Market Parameters Depth of Search How deeply into the pool of candidates departments are willing to go. Specify as max depth. 10 to 30 [3 levels] There are 3*2*3*3*2*2*3 = 648 parameter sets; 30 draws from each set  19,440 observations

31. Simulation Setup A single simulation run • Initial Conditions • 100 departments • Size distributed normally with mean of 25 std of 12 and an initial floor of 10. This is the resource-based target size for departments. • Faculty quality is distributed normally (N(0,1)) • Age is initially distributed uniformly from 0 to 40 (starting with a distribution means that retirements don’t go in waves) • Parameter Settings • Hiring curve: Medium • Student Production: 0.06 (~150 applicants per year) • Student-Faculty Quality Correlation: 0.67 • Disagreement on applicant quality: 0.60 • Disagreement on department quality: 0.1 • Hiring Rounds: 4 • Depth of Search: 20

32. Simulation Setup A single simulation run Market Size: • Over the first 10 years: • 66 to 104 positions advertised • 147 to 169 students on the market • 59 to 72 people were hired each year

33. Simulation Setup A single simulation run Distribution of size over time

34. Simulation Setup A single simulation run Correlation between final size and target size Quality > Mean + 1std Quality < Mean + 1std Target Equality Final Size Target Size

35. Simulation Setup A single simulation run Correlation of Size and Quality over time Burris reports the correlation between size and prestige as 0.63

36. Simulation Setup A single simulation run Correlation of Quality 10 years prior

37. High Competition Low Competition Results All results are presented around the competitive field: Disagreement on Candidate Quality 0.3 0.6 0.9 10 20 Depth of Search 30

38. Results Market Clearing: proportion of jobs that are filled Calculated for year y=100

39. Results Size & Quality: Department Size Calculated for year y=100

40. Results Size & Quality: Average Department Quality Calculated at final year ( y=100)

41. Results Size & Quality: Correlation of Size and Quality Calculated at final year ( y=100)

42. Results Quality Stability: 10 Year Correlation of Quality Calculated at final year ( y=100)

43. Results Department Features Summary Calculated at final year ( y=100)

44. Results Academic Castes? • The production and hiring of PhDs generates an exchange network, connecting the “sending” department to the hiring department. • I record this network for all hires in the last 10 years of the simulation history, and construct two measures: • a) The network centralization score • b) The correlation between network centrality & quality & size. • 10 years is conservative  all of the centralization effects I describe below are stronger if you limit the network to the last 5 years (which is closer to what people have done in the literature).

45. Results Academic Castes: Process Expectations A basic feedback process seems to be operating, and that should lead to a highly centralized and stable system.

46. Results Academic Castes? For what follows, working within one region of the parameter space Disagreement on Candidate Quality Depth of Search A preliminary regression over the entire space shows that hiring rates & quality correlation matter most for centralization

47. Results Academic Castes: Network Centralization by Quality Correlation & Job Openings Bonacich Centrality

48. Results Academic Castes: Network Centralization by Quality Correlation & Job Openings Degree Centralization

49. Results Academic Castes: Correlation of Centrality & Department Size Bonacich Centrality

50. Results Academic Castes: Correlation of Centrality & Department Size Degree Centrality