estimating the causal impact of financial aid programs on student retention rates n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates PowerPoint Presentation
Download Presentation
Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates

Loading in 2 Seconds...

play fullscreen
1 / 39

Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates - PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on

Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates. A Non-Technical Introduction to RD and ITS Designs. Michigan Association for Institutional Research Nov. 6-8, 2013 Grand Rapids, MI. Reuben Ternes Oakland University OIRA ternes@oakland.edu.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates' - triage


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
estimating the causal impact of financial aid programs on student retention rates

Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates

A Non-Technical Introduction to RD and ITS Designs

Michigan Association for Institutional Research

Nov. 6-8, 2013

Grand Rapids, MI

Reuben Ternes

Oakland University

OIRA

ternes@oakland.edu

intro retention

Increased Scrutiny Over Retention Rates at OU

    • Rates now tied to funding
    • Historical retention rates between 70% and 75%
    • President requested a review of the impact that our financial aid policies had on retention rates.
    • This presentation details the results and methods used to conduct that review.
Intro: Retention
presentation outline

Intro: Retention

  • Section I: Policy Overview
  • Section II: Regression Discontinuity Designs
  • Section III: Interrupted Time Series Designs
Presentation Outline
i policy overview

OU aid is very transparent

  • Published criteria given to public
  • This presentation covers need-based aid only
  • Two different types of need aid at OU:
    • 1) Academic/Housing grant – Must have a HSGPA of a 3.0 or higher and an ACT of 21 or more. Must demonstrate need. Up to $8k.
    • 2) 100% Tuition Award – Covers gap between tuition and EFC after all other aid is calculated.
I – Policy Overview
i policy overview1

Academic/Housing Grant: Contains very hard criteria for inclusion. Suitable for RD analysis.

  • 100% Tuition Award: Sudden onset. Suitable for ITS analysis.
I – Policy Overview
ii rd overview

RDD = Regression Discontinuity Design

  • How can you assess program effectiveness if you separate participants based on expected performance?
  • Specifically: How can we be sure that our financial aid program improves retention rates, when the aid that we give is confounded with prior student ability?
    • (ACT scores & HS GPAs are positively correlated with retention rates)
II – RD Overview
ii rd basic logic

Basic Logic – When you are looking at an outcome and relating that outcome to some measure, but have a sudden change of group status, a discontinuity will be visible at that change if group membership has an impact on your outcome.

II – RD Basic Logic
ii rd logic

People with very similar ‘assessment’ scores are similar.

    • If group membership is controlled by a strict cut-off
    • and similar people have very different experiences
    • then a gap should be visible if the group membership impacts the outcome
  • The above logic is extremely useful and powerful
  • The mathematics of RD is about establishing unbiased treatment effects similar to random assignment.
II – RD Logic
ii rd visuals

The logic of RD is simple & intuitive

  • Often times, visuals are enough to confirm causation and guide policy decisions
    • (i.e. no regression is actually required)
    • Visuals are very convincing to many people
  • Unlike standard regression techniques, RD’s logic is inherently causative, not correlative.
II – RD Visuals
ii rd in action

Do our grants cause student to stay at the university?

  • If so, can we guess how many students it impacts?
II – RD in Action
ii rd in action step 1

First, establish that nothing strange is happening at cut-off for students that do not receive the grant.

    • (i.e. control condition)
    • Technically, you don’t need this step, but it does help convince some people.
  • (What you are about to look at is a graph of retention rates by ACT score. Each ACT score has its own retention rate, which is the collective rate for each student with that score. Each ACT score is composed of hundreds of students).
II – RD in Action, Step 1
ii rd in action step 2

Now examine what happens at the cut-off for students that do receive the grant.

  • Notice the large, obvious gap!
    • This discontinuity is where the method gets its name from.
II – RD in Action, Step 2
rd in action step 3

Now graph them together, so people can see them in relation to each other.

  • Ignore the behavior near the extreme tails. It has no real meaning within an RD framework.
    • Also, the N is smaller for extreme ACT scores.
RD in Action, Step 3
ii policy changes

The evidence in this case is very clear.

  • Our need-based aid improved retention rates
    • Estimating by how much is harder.
    • For the students near the cut-off, retention rates improved by about 10%.
  • Based on this data, we reduced the criteria to obtain the grant, so more students would qualify.
    • We will continue to monitor the progress of these students now that the criteria has changed.
    • No theoretical reason for the impact to be equivalent across all potential ACT scores.
II – Policy Changes
ii rd caveats

RD is not better than random assignment

    • It can only produce estimates near the cut-off.
      • The effect size in the tails is unknown
    • Obtaining unbiased treatment estimates is still a challenge
      • Must model the functional relationship exactly – linear, quadratic, cubic, something else?
      • ‘Fuzzy’ RDs, where the cut-off criteria is mostly adhered to, present additional challenges
II – RD Caveats
iii its

ITS = Interrupted Time Series Design

  • Logic: Essentially it is an extension of ‘before’ and ‘after’.
III - ITS
how its shows causation

Control

Treatment

Control

Treatment

How ITS Shows Causation
its causation

Subjecting the same group to different conditions, repeatedly, creates a very strong case for causation.

    • If differences are found, causation is only threatened if something else varies alongside the timing of the treatment/control condition.
    • In most cases, it is very unlikely anything else will vary in such a manner.
ITS & Causation
its in educational research

Opportunities with an ideal ITS in educational research are very rare.

    • We rarely can subject the same sample to repeated conditions.
    • Mostly, we get ‘before and after’ conditions, often with different populations (i.e. years)
ITS in Educational Research
its still valuable

But the basic principles of ITS are still valuable!

    • In many cases a large sample size mitigates most potential issues of not using the same sample.
    • Though repeated measurement is best, a single measurement can still be persuasive.
ITS – Still Valuable!
background
In 2005, Institution X created Remedial Math 3
    • Designed to address deficiencies with some students in Remedial Math 2.
    • New placement scores were developed for several courses.
  • Because this policy change happened suddenly, it is possible to analyze data from this time period as a ‘natural experiment,’ comparing students from before the policy change (2004) and those after the policy change (2005).
Background
percentage of students that complete the entire remedial math sequence
Percentage of Students that Complete the Entire Remedial Math Sequence

Two Year Completion Rate

Math 3

Implemented Here

These Students

Take Math 2

These Students

Take Math 3

There was a significant drop in 2-year completion rates for Math 1 just after

the implementation of the new Remedial Math 3 course. The sample size for each year is close to 1,000

its example 22

Highest Year-to-Year Variance (pre-2012): 3.1%

  • Average Year-to-Year Variance (pre-2012): 1.5%
  • 2011-2012 Variance: 7.8%
    • That is more than double the previous high, and five times more than average!
  • Q: What Happened?
    • A: We changed our entire approach to first year advising and created a first year advising center (FYAC)
    • Classic example of potential ITS.
      • Strong and sudden onset of a novel environment
ITS – Example 2
its example 23

There are more internal threats to validity with ITS than there are with RD.

    • In using this technique, you will still probably have to think about plausible alternatives and see if you rule them out.
  • Two plausible explanations still exist in this example:
    • 1) Regression to the mean (2011 had a very low retention rate, so year-to-year variation could be artificially high).
    • 2) The 2012 cohort also had the highest incoming academic profile.
  • Could this combination explain the results?
ITS – Example 2
its example 24

Simple regression estimates suggest that some of the variation is explained by the alternative explanations.

    • About 3% is still left unexplained though, which bodes well for our FYAC (and our decision to create it).
  • The explanation can be graphed visually!
    • Graph displays 2012 cohort, 2011 cohort, and historical rates for comparison.
    • The tails of the distribution have low N. Generalize here only with great caution.
ITS – Example 2
its example 3

Two different types of need aid at OU:

    • 1) Academic/Housing grant – Must have a HSGPA of a 3.0 or higher and an ACT of 21 or more. Must demonstrate need. Up to $8k.
    • 2) 100% Tuition Award – Covers gap between tuition and EFC after all other aid is calculated.
  • We will now focus on #2, the 100% tuition award.
  • Visually, you do not have to present ITS as a timeline – there are other ways to think of it, such as comparing historical rates to current rates.
ITS – Example 3
its example 31

What impact did our 100% tuition award have?

    • Analyze data by Expected Family Contributions
      • In theory, students with high EFCs won’t get much award money, and probably don’t need as much.
      • So impact should be different across EFC groups.
    • Plot retention rates by EFC for historical rates, and then under the 100% tuition award.
ITS – Example 3
its example 32

Possible Interpretations

    • The results were somewhat unexpected, as very low EFC students did not benefit from the aid.
    • Possibly the aid just wasn’t enough to impact their retention decision, because the amount of money just won’t cover living expenses.
      • As EFC rises, the aid, though less on average, becomes more of a decision factor for these families.
  • Threats to Validity
    • Only one control year.
    • The economic recession is difficult to model, could cause a great deal of variance in unexpected ways.
ITS – Example 3
its conclusions

In general, ITS is easier to use than RD.

    • But it generally has more threats to validity.
    • It’s a great tool to try and estimate causality, but it is far from perfect.
ITS - Conclusions