Inference by College Admission Departments: Bayesian or Cursed?

1. Inference by College Admission Departments:Bayesian or Cursed? Michael Conlin Michigan State University Stacy Dickert-Conlin Michigan State University Cornell University– October 2009

2. Optional SAT I Policy Colleges with optional SAT I policy allow applicants to submit their SAT I score but do not require them to submit their score. (As of Spring 2009, more than 800 colleges have SAT- or ACT- optional policies.)

3. Research Question What does the admission department at the college infer when an applicant chooses not to submit her SAT I score? This inference influences whether or not the school accepts the applicant.

4. Bayesian vrs. Cursed Equilibrium Most game theoretic papers assume the inference is based on Bayes Rule. Eyster & Rabin (Econometrica 2005) document extensive experimental evidence and develop a simple model of how players underestimate the relationship between other players’ actions and their private information. They call this psychologically motivated equilibrium concept a cursed equilibrium.

5. Eyster and Rabin (page 1633) “The primary motivation for defining cursed equilibrium is not based on learning or any other foundational justification, but rather on it pragmatic advantages as a powerful empirical tool to parsimoniously explain data in a variety of context.”

6. Why is this important? Think of any private information game – including those resulting in adverse selection, signaling and screening. Most of these games apply Bayesian Nash Equilibria to predict outcome. The Bayesian Nash Equilibrium Concept assumes that players apply Bayes Rule at all information sets that are reached with positive probability. Expected outcomes could change significantly if this were not the case.

7. Example using Spence Education Model Nature Low Productivity Worker High Productivity Worker e=0 e=0 e=16 Firm e=12 e=12 e=16 Suppose a separating equilibrium exists where the low productivity type gets 0 years of education and the high productivity type gets 16 years of education.

8. Voluntary Disclosure/ Verifiable Cheap Talk Example • Student i has the following probability distribution in term of SAT I scores. • When disclosure is costless, Bayesian Nash Equilibrium results in every type except the worst disclosing and the worst being indifferent between disclosing and not disclosing.

9. Voluntary Disclosure / Verifiable Cheap Talk Models • Comments: • Distribution depends on student characteristics that are observable to the school such as high school GPA. • With positive disclosure costs, the “unraveling” is not complete and only the types with the lower SAT I scores do not disclose. • Assumptions: • Common Knowledge. • Colleges use Bayesian Updating to Infer SAT I Score of those who do not Submit/Disclose • Colleges’ incentives to admit an applicant is only a function of his/her actual SAT I score (not whether the applicant submits the score)

10. Eyster and Rabin applied to Voluntary Disclosure / Verifiable Cheap Talk College correctly predicts the distribution of the applicant’s actions but underestimates the degree these actions are correlated with the applicant’s SATI score (i.e., private information). “Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose that his/her expected SAT I score is 1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170 “Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant doesn’t disclose that his/her expected SAT I score is (.4)1170+ (1-.4) [(1100(.3)+1000(.1))/.4] = 1113

11. Why is this important in the context of SATI optional policy? If, (1) a college underestimates the relationship between an applicant’s decision to not submit and their SATI score (i.e., private information) and (2) certain types of applicants are less likely to submit, then the SATI optional policy is likely to affect the demographic characteristics of the student body.

12. Overview of Environment SAT EXAMS: College Board administers SAT I & SAT II: Subject exams 7 times/year between October and June. SAT I was a 2-part verbal and math exam. There are 20 SAT II exams. EARLY DECISION: Submit application by mid-November & sign a written agreement (with guidance counselor) that states that upon admittance the applicant will attend if she can “afford it” (agreement not legally binding). Notified in December of acceptance. The deadline for regular admission applications is January 1st and these applicants are notified of acceptance or denial between March and April. An accepted applicant that chooses to enroll fills out enrollment forms and makes a deposit by May 1st.

13. Applicant’s Early Decision and SAT Submission Decisions Applicant i’s expected utility conditional on attending the college is μ(Zi, εap, εen)= μ(Zi) + εap + εen where, μ(Zi)captures individual–specific preferences that depends on observable variables (Zi), εap represents unobservable individual–specific preferences known to the applicant at the time she submits the application, and εen represents unobservable individual–specific preferences not known to the applicant until she makes the enrollment decision. • Applicant i’s expected utility if doesn’t attend UR if apply regular admission UR-C if apply early decision

14. Applicant’s Early Decision and SAT Submission Decisions Applicant i’s expected utility from not applying early decision to the college and submitting SATI scores is: where, is the applicant’s expectation of the probability she will attend, εs represents the “cost” of submitting which is known to the applicant at the time she submits application, and K represents fixed cost of applying.

16. Applicant’s Early Decision and SAT Submission Decisions Assuming UR=0 (a normalization) and E(εen)=0, an applicant applies early decision and does not submit if :

17. CONDITION 1 CONDITION 2 CONDITION 3 is not binding

18. College’s Objective Function To account for the college’s concern for the quality of its current and future students and the understanding that future student quality depends on the college’s ranking, we allow the college’s objective function to depend on the perceived ability of the incoming students, the “reported” ability of these students, the demographic characteristics of the student body and yield rate.

19. College’s Objective Function where, ΠP(ZP) captures how the college’s payoff is affected by the perceived student body’s ability level, ΠR(ZR) captures how the college’s payoff is affected by the reported student body’s ability level, ΠD(ZD) captures how the college’s payoff is affected by the demographic characteristics of the student body, f(YR) captures how the yield rate affects the college’s payoff

20. College’s Acceptance Decision College accepts applicant i if: where, denotes the expected probability an accepted applicant i enrolls where k equals ed (ned) if applicant i applies early decision (regular decision) and l equals s (ns) if the applicant i submits (does not submit) her SATI score, Z+i, Z-i, Zri are expected characteristics of student body if applicant i is accepted and enrolls, is accepted but does not enroll, and is rejected.

21. College’s Acceptance Decision The probability applicant i is accepted if she applies early and does not submit is: with the expected student body characteristics being the same if applicant i is accepted but chooses not to enroll (Z-i) or if applicant i is rejected (Zri).

22. Applicant’s Enrollment Decision • An accepted applicant i who does not apply early decision will enroll if: μ(Zi) + εap + εen > UR • An accepted applicant i who does apply early decision will enroll if: μ(Zi) + εap + εen > UR – C

23. Applicant’s Enrollment Decision The probability an early decision applicant i, who does not submit, enrolls conditional on being accepted is: Pe(Zi,e,ns)= Prob[εap+εen>-μ(Zi)-C, Conditions 1,2&3 hold] / Prob[Conditions 1,2&3 hold].

24. Generating a Likelihood Function • Parameterize the cursed equilibrium concept proposed by Eyster and Rabin; • Make assumptions about the applicants’ expectations of attending and the college’s expectations of enrollment; • Impose functional form restrictions on the college’s and applicants’ objective functions; • Make distributional assumptions for the random components.

25. Parameterizing cursed equilibrium We assume college’s belief of applicant i’s SAT score if she does not submit is SATi,un+ (1- )SATi,cond where, SATi,un is the belief of applicant i’s score if the college does not condition on her choosing not to submit and SATi,cond is the belief if the college does condition on her choosing not to submit.

26. Parameterizing Eyster and Rabin’s cursed equilibrium concept • We construct SATi,un • regress SATI scores for applicants who submit on their observable characteristics. • use the resulting coefficient estimates to construct fitted values for all applicants, i, who do not submit. • assume the fitted value for applicant i is what would be inferred if the college believes that applicant i’s decision to not submit did not depend on her actual SATI score. • We assume SATi,cond is applicant i’s actual SATI score.

27. Eyster and Rabin applied to Voluntary Disclosure / Verifiable Cheap Talk College correctly predicts the distribution of the other players’ actions but underestimates the degree these actions are correlated with the other players’ private information. “Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose that his/her expected SAT I score is 1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170 “Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant doesn’t disclose that his/her expected SAT I score is (χ)SATun+ (1-χ)SATcond = (.4)1170 + (1-.4) [(1100(.3)+1000(.1))/.4] = 1113

28. Applicants’ expectations of attending

29. College’s expectations of enrollment Assume college uses the information they have on each applicant to accurately predict their enrollment probabilities. For those who submit, For applicants who chose not to submit their SATI scores, the college’s expected probability of enrollment is similar to the predicted probability except the college infers an SATI score of SATi,un+ (1- )SATi,cond while the predicted probability, Pe(Z,k,l), is based on their actual score.

30. Functional Form Restrictions μ(Zi) = βμ•Zi f(YRai) = βYRYRai, f(YRri) = βYRYRri ΠP(Z+iP)= βP•Z+iP , ΠP(Z-iP)= βP•Z-iP ΠR(Z+iR)= βR•Z+iR , ΠR(Z-iR)= βR•Z-iRand ΠD(Z+iD)= βD•Z+iD , ΠD(Z-iD)= βD•Z-iD

31. Functional Form Restrictions Assuming the college’s expectations of Z-ij, YRri, and Nsb do not vary across applicants, these functional form restrictions result in: where βP’= βP/(Nsb+1), βR’= βR/(Nsb+1), βD’=βD/(Nsb+1),βYR’= βYR YRri/(N+1) and βint is a function of Z-ij (for all j), YRri, Nsb and N.

32. Functional Form Restrictions Note that not all the parameters are identified. For example, suppose high school GPA affects both perceived quality and ranking, then can only identify βPGPA+βRGPA. We can identify βP and βR associated with the SATI scores because some applicants submit while others do not.

33. Distributional Assumptions εap, εen, εs and εqi are standard normal and independent.

34. Likelihood Function Goal is to estimate a vector of structural parameters θ = { , βμ, βe,βks, βksat,βint, βP’, βR’, βD’,βYR’ and C}. Derive parametric expressions, as a function of observed data and structural parameters, for • the probabilities an applicant applies early decision and/or submits her SATI score • the probability the college accepts an applicant • the probability an accepted applicant enrolls. Expressed with standard normal and standard multivariate normal distributions.

35. Data • Application data for 2 liberal arts schools in the north east • Each with approximately 1800 students enrolled. • Both report a typical SAT I score in the upper 1200s/1600 • College X: 2 years ≈ 5 years after the optional policy was instituted. • College Y the year after the optional policy was instituted.

36. College Board Data • SAT scores for those who elected not to submit them to the college. • Student Descriptive Questionnaire (SDQ) • SAT II Scores • Self Reported income • High school GPA • High school activities

37. Descriptive Statistics for College X Early Decision Applicants more likely accepted Early Decision Applicants more likely to enroll Applicants that apply early decision and don’t submit have lower SATI scores Number of Applicants 324 122 5216 895

38. Descriptive Statistics for College X Female applicants from private high school less likely to submit Applicants who apply early are weaker students

39. Descriptive Statistics for College X Legacies and high income applicants are more likely to apply early decision

40. Descriptive Statistics for College Y Similar to College X except: • Early Decision applicants are less likely to be accepted (~.65 compared to ~.85 for College X) and enroll conditional on acceptance (~.92 compared to ~.98 for College X) • Measure of academic performance are slightly worse for applicants who do not submit their SAT I scores.

41. Results from reduced form papers (Dickert-Conlin & Chapman; Conlin,Dickert-Conlin & Chapman) • Applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics.

42. Results from reduced form papers SATIun

43. Results from reduced form papers (Dickert-Conlin & Chapman; Conlin,Dickert-Conlin & Chapman) • Applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics. • College admission departments are behaving strategically by rewarding applicants who do submit their SAT I scores when their scores will raise the college’s average SAT I score reported to U.S. News and World Report and rewarding applicants who do not submit when their SAT I scores will lower the college’s reported score.

44. Results from reduced form papers Applicants who submit their SAT I score are less likely to be accepted by College X if their SAT I score is below 1388 and are more likely to be accepted if their score is above 1388. Average SAT I score at College X is 1281. Set of regressors include SATII score, ACT score, private high school, female, HS GPA, HS class rank, family income bracket, legacy, early decision indicator, race, geographic location, HS extracurricular activities. Subsequent measures of college performance are not correlated with scores and submission decisions in a similar manner.

45. Table 2: Structural Parameter Estimates College X has more years of experience with optional policy but admission director is relatively new. Expect probability of acceptance to increase with expectation of enrollment but difference in coefficient is worrisome. Estimates large as expected based on enrollment probabilities of early decision and early decision enrollment probability higher for College X.

46. Table 2 Structural Parameter Estimates Parameters of College’s Payoff Function Utility Parameters

47. Table 2 Structural Parameter Estimates

48. Table 2 Structural Parameter Estimates

49. Table 3: Model Fit College X

50. Table 3: Model Fit College Y Model fits the other observables well in terms of whose accepted and those who enrolled.