Master’s Thesis Sarah M. Hunt September 2013

1. The Impact of Trajectory Prediction Uncertainty on Reliance Strategy and Trust Attitude in an Automated Air Traffic Management Environment Master’sThesis Sarah M. Hunt September 2013

2. Identifying necessary controller characteristics for using dynamic, predictive automation systems from the NextGen environment. Introduction

3. The Environment Perfect Estimation With Dynamic Predictive systems, a controller is required to relinquish and regain the use of their tools according to current automation state. Relinquish Initial Under Estimation Over Estimation Regain Range of Usefulness

4. The Controllers The Ideal NextGen Controller Recalibrates based on real time data, relinquishing and regaining use of the tool as accuracy fluctuates. Uses a system to its full capability and only for its intended purpose. Maintains stable trust in the overall performance of the system which is largely unaffected by environmentally driven errors.

5. The Development of Ideal Behavior “Behaviors result from intentions and that intentions are a function of attitudes.” (Lee and See, 2004, p. 53). Temporal Specificity Initial Calibration Subjective Norm & Perceived Behavioral Control Ajzen & Fishbein, 1991. Theory of Planned Behavior

6. Identifying Ideal Controllers Calibration & Temporal Specificity Reliance Strategy Underlying Trust Subjective Trust Assessment Jian, Bisantz and Dury’s Human-Automation Trust Scale. Does this scale measure underlying trust in an air traffic management environment? • Objectively assessing how controllers used the automation’s assistance over time. Can we measure these in an effective way to identify Calibration & Temporal Specificity? Together do these suggest related but distinct constructs?

7. Methods

8. Design & Procedures Environment, Procedures & Goals Simulated Airspace • Environment: En Route Airspace over Atlanta with one High and one Low sector using historical based simulated traffic. • Procedures: 2 Parallel, independent simulations conducted simultaneously. • 12 runs, 55 minutes long. • Goal: Schedule based operations- operators working to deliver aircraft to ERLIN within +/- 20 seconds.

9. Tools Mid Term NextGen Environment Management Tools • Assumption of accurate surveillance and advanced decision support tools. • Metering based tools available, including Trial Planning and Delay Times. • TS (trial plan speed) advisory only gave speed suggestions to the controllers. TS Action Sequence

10. Variables for an Uncertain Environment IV: Error Condition DV: Objective & Subjective Delay Tag Time (DTT): early / late delay in seconds at the time of the TS. Difference in Issued vs. Advised Speed (DIA): operator’s modification on speed advisory in knots. Likert Score: scoring on a Human-Automation Trust Scale with 7 statements both positive and negative (Jian et al, 2000). • Six levels combining Wind Forecast and Aircraft Performance errors in the automation's calculations.

11. Two analysis designs results

12. Design 1: Reliance Strategy through Objective Data Looking for Temporal Specificity: Change in Reliance Strategy over Time Conditions with Significant DTT, DIA Correlations: RM & RL • NN, RR, RL, LR, RM, LL • In the Red conditions, DTT (automation state) predicts DIA (reliance strategy)

13. Design 2: Construct Validity of the Human-Automation Trust Scale

14. Design 2.1: Differences in Statement Types 2X6 Repeated Measures ANOVA • Primary: All conditions had significant (p<.005) differences in the mean Likert Scores of Positive and Negative Statement types. • Follow Up: Post Hoc pairwise comparisons for Positive statements only, RL (M=4.8) significantly different (p<.005) from all conditions except RR.

15. Design 2.2: Internal Consistency Cronbach’s Alpha = .923 • Item N= 8 • Negative statements were reverse scaled (Suspicious, Wary, Harmful) • Removing Comfortable (the added keyword) would increase the score of this scale.

16. Discussion

17. Discussion: Distinct but Related Constructs Temporally Specific Reliance Strategy Mostly Stable Underlying Trust

18. Discussion: Identifying Ideal Controllers Calibration & Temporal Specificity Reliance Strategy Underlying Trust Subjective Trust Assessment Jian, Bisantz and Dury’s Human-Automation Trust Scale does measure a single trust construct in an ATM environment. Using an objective measure requiring interaction that is directly related to automation state can uncover Calibration & Temporal Specificity. Measured together, these factors approach understanding the whole controller as they train on dynamic & predictive systems. Application: Measure to identify improperly calibrated controllers who tend to lack temporal specificity in the training phase before safety is compromised.

19. Discussion: Implications & Limitations Measuring Temporal Specificity & Calibration Measuring Attitude and Intent Trust Scale Modifications: Both proper counterbalancing and factor analysis are suggested for the Human-Automation Trust scale. Measuring Additional Factors: Subjective Norm and Perceived Behavioral Control will also affect Intent & Calibration. • New tools = new measurement for reliance strategy • Multiple measures would give a more complete picture of reliance strategy. • Themore the controllers understand about potential errors, the more easily they will recalibrate. Limitation: The sample size was very small, and while they were subject matter experts, more data should be collected for better generalizability.

20. Acknowledgements Thanks To: Dr. Kevin Jordan Dr. Lynne Martin Dr. Sean Laraway Joey Mercer, M.S. Dr. Thomas Prevot The team of the Trajectory Prediction Uncertainty Experiment

21. The Impact of Trajectory Prediction Uncertainty on Reliance Strategy and Trust Attitude in an Automated Air Traffic Management Environment Master’s Thesis Sarah M. Hunt September 2013

22. Appendix

23. Error Structure Wind Error Aircraft Performance Error

24. Scale Presentation in Lime

25. ANOVA Data Tables

26. 2X6 Repeated Measures ANOVA • There was a statistically significant interaction between Trust Statement Type and Error Condition on mean Likert Score, F(5, 40) = 11.902, p =.002, partial η2 = .598. • There was no significant main effect for Error Condition, F(5, 40) = .409, p =.839, partial η2 = .049. • However a significant main effect was observed for Trust Statement Type, F(1, 8) = .729.852, p <.001, partial η2 = .989. Pairwise comparisons for Trust Statement Type show m=3.907, SE =.145, p<.001, 95% CI [3.574, 4.241].

27. Procedures • Twelve 55 minute runs with errors of varying degrees and breaks in between.

28. Implications & Limitations Additional Factors Limitations Small sample size- with only 3 controllers contributing to the final analysis. The controllers were retired- how generalizable are they? • This experiment assumed the Subjective Norm was stable and driving Intent. • In a real world environment the normative beliefs of a facilities' crew could greatly influence use of the automation.

29. Design 2.2: Correlated Statements Spearman’s Rho Testing Correlations between 8 Statements Significant Correlation Pairs- Comfortable Removed for Clarity • Significant Negative Correlations were shown between Positive & Negative statements. • 19 of 56 possible correlations were significant (p<.05) • Harmful, Reliable and Trustworthy were highly correlated with all other keyword. Suspicious & Safe Solutions were not significantly correlated, suggesting suspicion can be maintained while judging the automation’s solutions currently safe.

30. Teasing out Relevant Factors to ATM Environments Environmental Constraints (Over Time) Beliefs Multiple factors influencing Reliance Strategy regardless of Trust Attitude. Knowledge Trust Attitude RELIANCE STRATEGY Intent Affective Factors Subjective Norms Instead of a one to one mapping, Trust Attitude and Reliance Strategy are Related but Distinct Constructs. Personal Behavioral Control Trust (Underlying Good Faith), Other People’s Opinions & What a Controller Believes They Can Achieve