The Cycling of a Decision Threshold: A System Dynamics Model of the Taylor Russell Diagram

1. Elise Axelrad Weaver, Ph.D. and George Richardson, Ph.D. Center for Policy Research April 20, 2001 The Cycling of a Decision Threshold: A System Dynamics Model of the Taylor Russell Diagram

2. The Cycling of Decision Thresholds • In his book, Hammond (1996) presents the following ideas: • Any decision threshold based on a statistically uncertain measure will inevitably yield some error and injustice in policy outcomes (a duality of error: false positives and false negatives) • Oscillations in public and professional attitudes (with implicit policy thresholds) exist: • Schlesinger’s (1986) proposal of “regular oscillations” in the dominance of political parties • Oscillations between cautious conservatism and risky innovation in bridge design, as “the accumulation of successful experience” makes designers bold until a “collosal failure” takes everyone by surprise • Cycles tend to be around 30 years long, across decision domains • Hammond suggests that oscillations may be a result of the duality of error

3. Hammond’s exhortation: “For if such oscillations can be shown to exist, and if they can be shown to have a definite period...then we have at hand not only a means for predicting our future political climate far in advance, but an important phenomenon that strongly invites, indeed, demands, analysis and interpretation.” We propose a system dynamics model to represent and explore Hammond’s idea in a rigorous way

4. Another Example of a Reversal in Policy Formation Use of SAT testing in admissions 1967 University of California began using SAT scores in admissions decisions 1999 University of California faculty voted their preference to exclude SAT scores from admissions decisions 2001 Richard Atkinson, University of California President, publicly advocates removing SATs from admissions.

5. Example of a Decision Threshold for Policy Formation • Illustration: • Students are admitted to an academic program • partially according to SAT score set at a given • threshold • The SAT score is used to predict academic success • The GPA at graduation is the measure of true academic success

6. Taylor Russell Diagramr = .5, cutoff = 50 False - Graduates Non-Graduates False + Negative on test Positive on test Decision Threshold

7. Taylor Russell Diagramr = .5, cutoff = 80 False - Graduates Non-Graduates False + Negative on test Positive on test Decision Threshold

8. Taylor Russell Diagramr = .5, cutoff = 20 False - Graduates Non-Graduates False + Negative on test Positive on test Decision Threshold

9. Taylor Russell DiagramHighly Certain Test (r = .99) False - Graduates Non-Graduates False + Negative on test Positive on test Decision Threshold

10. Stakeholders in the Duality of Error Constituency Concerned with Unfair Disadvantage: They want to reduce the number of individuals for whom the decision to accept was falsely negative: high potential for success but unacceptable SAT scores Constituency Concerned with Maintaining Standards: They want to reduce the number of individuals for whom the decision to accept was falsely positive: low potential for success but acceptable SAT scores

11. Duality of Error Due to the Decision Threshold False - False + Decision Threshold

12. Decision Threshold as a Stock (accumulating increases and decreases)

13. Decision Threshold Responding to Stakeholder Pressure Disadvantaged Constituency (DC) High Standards Constituency (HSC)

14. Cycling of Policy Threshold: Historic Discontent Disadvantaged Constituency (DC) High Standards Constituency (HSC)

15. Cycling of Policy Threshold: Key Parameters Disadvantaged Constituency (DC) • For both constituencies: • Tolerated Number of False Cases • Relative Weight on History • Time to Respond to Pressure • Threshold Change per Unit Pressure High Standards Constituency (HSC)

16. Cycling of Policy Threshold: More Key Parameters • Time Constants (Forgetting, Retention) • Initial value of Threshold • Lookup functions (correlation) • Dissatisfaction per Error

17. Unexplored Parameters Lookup Function • translation of threshold choice to number of errors • involves correlation, thresholds for judgment and success • currently set at r = 0.7, variable threshold for judgment, and threshold for success fixed. Pace of Error Generation and Error Detection • Baseline model assumes that in every month, there is an assessment of number of errors due to threshold choice Threshold Associated with “Success” • Baseline model assumes a fixed threshold of “success”

18. Graph of Cycling of Policy Threshold SAT Score Baseline Case

19. Threshold Change per Unit Pressure Time to Respond to Pressure - represents policy makers’ responsiveness to constituent pressure: more responsiveness means wider amplitude and very slightly lower frequency

20. Initial Threshold Value - represents policy makers’ initial setting for decision threshold: no long term effect

21. Tolerated Number of False Cases - represents constituents’ sensitivity to errors not a key difference in the long run for either frequency or amplitude

22. Time Constant for Forgetting - represents time it takes for an error to dissipate from constituent memory more time to forget lowers frequency and raises amplitude

23. Relative Weight of History - represents constituents’ memory or weighting for accumulated past errors relative to current error more weight on history means lower frequency and higher amplitude of cycling; no weight on history means no cycling at all.

24. Different Weight on History for Each Constituency - represents differential weight on history by constituents - if one group puts more weight on history, it doesn’t change the center of the cycling, but it does reduce the amplitude.

25. Key Parameter Summary • Parameters not affecting frequency: • initial decision threshold • policy maker sensitivity: increases amplitude,but doesn’t affect frequency much in the long run • tolerated number of errors by constituents: does not have much effect on frequency or amplitude • Parameters affecting frequency • relative weight on history & time to forget: longer memories lower frequency and raise amplitude.

26. Ground the conceptual model in data from one or more institutions (especially look for cases where false negatives could be detected) • Incorporate the effects of correlations between the judgment and success measures, time to generate and detect errors, and the threshold for success • Explore and operationalize the conversion variables (from errors to stakeholder pressure to changes in threshold) • Consider whether there is any impact of true negatives and true positives on cycling • Explore limit cycle and open loop structure of model Next Steps

27. A systems dynamics model is under development. • According to Hammond (1996), any uncertain test where a threshold is used as a policy decision tool leads to unavoidable injustice to some constituency. • The pressure on the decision threshold from stakeholders representing the false positives and the false negatives will oppose. • These opposing pressures will cause a cycling of the decision threshold over time. • A key parameter affecting frequency and amplitude of cycling is constituent weighting of past errors. Summary