Direct metrics of driver performance

1. Direct metrics of driver performance Johan Engström Volvo Technology Corporation Driver Metrics Workshop Ottawa, October 2-3, 2006

2. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

3. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

4. Research conducted 2003-2006 in the HASTE and AIDE EU-funded projects General objective of the studies: Investigate systematically the effects of visual and cognitive load on driving performance -> define metrics for IVIS safety evaluation Data collected in simulators (of varying grade) and field (HASTE) Further analysed in AIDE Work reported here performed in collaboration with VTI (Swedish National Transport Research Institute) HASTE Research in HASTE and AIDE on performance metrics

5. Collected in HASTE WP2 during 2003-2004 9 parallel studies at different sites in Europe and Canada Same general methodology and experimental design Varied mainly with respect to test set-up (desktop simulator, meduim-high-fidelity simulators and field trials) Secondary tasks: The HASTE WP2 data set Auditory Continuous Memory Task (aCMT) Auditory/cognitive: aCMT Visual: Arrows task 3 difficulty levels each

6. Three general driving scenarios: Motorway, Rural, and Urban Present analyses based on data from three sub-studies VTEC fixed-base simulator (rural and motorway) VTI moving-base simulator (rural and motorway) Volvo-VTI field study (instrumented Volvo S80 on motorway) The HASTE WP2 data set (cont’d) VTEC simulator VTI simulator Volvo S80

7. Visual and cognitive load have qualitatively different effects on driving… General result

8. Summary of results: Visual task

9. Summary of results: Cognitive task

10. Visual Visual diversion Steering hold Lane keeping error Large corrective steering movements Slowing down & increasing headway to compensate General conclusion: Visual and cognitive load have different effects Cognitive • Interference with attention selection mechanisms • Gaze concentrates to road centre • More visual control input than during normal driving • More active and precise steering • More accurate lane-keeping Reduced visual detection/ decision making

11. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

12. Metrics intended for task-based IVIS evaluation Types of metrics covered: Lane keeping Steering Eye movements Focus of this presentation

13. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

14. Example data: Straight driving, rural road, VTEC simulator

15. Frequency analysis Individual subjects All subjects

16. Focus here Non-normal distribution & too few instances -> difficult to use for task-based evaluation Less sensitive than position-based metrics and yield roughly similar results -> no obvious advantage for present purposes Common types of lane keeping metrics e.g. proportion of lane exceedences (LANEX) e.g. standard deviation of lane position (SDLP) Position-based e.g. mean of TLC minima (MN_TLC) e.g. proportion of TLC minima < X s (PR_TLC) TLC (Time-to-line-crosssing) -based Continuous Event-based

17. Operational definition (AIDE D2.2.5 – Östlund et al. 2006): ”Standard deviation of lateral position data, high-pass filtered with a cut-off frequency of 0.1 Hz, where lateral position is defined as the average distance between the right side of the frontor rear right wheel and the inner (closest) edge of the right hand lane marking.” (Modified) Standard deviation of lane position (SDLP) (1)

18. SDLP depenency on data duration High-pass filtering needed to overcome this problem (Östlund et al., 2006)

19. Representative results from HASTE on SDLP VTEC simulator, rural road Cognitive task Visual task

20. Advantages Easy to measure, at least in the simulator (feasible also in the field using off-the-shelf lane-tracking systems) Straightforward general interpretation as performance metric Disadvantages Only moderately sensitivite to secondary task task load Strongly sensitive to environment factors (e.g. curvature, lane width) Sensitive to discontinuities due to lane changes and exceedences Relation to crash data Open issue – no strong direct evidence of causal relation between increased SDLP and crash risk (however, indirect evidence via visual distraction) (M)SDLP pros and cons

21. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

22. Example data: Straight driving, rural road, VTEC simulator

23. Standard deviation of steering wheel angle High frequency steering – 3 versions Steering entropy – 2 versions (Boer, 2000;Boer, 2005) Steering wheel reversal rate – 2 versions (HASTE version; Modified version developed in AIDE, Markkula and Engström, 2006) Metrics investigated in AIDE

24. Results in sensitivity (effect size) – visual load All metrics fairly sensitive in all conditions except Standard Deviation Markkula and Engström (2006)

25. Results in sensitivity (effect size) – cognitive load Reversal Rate2 and Steering entropy most sensitive Markkula and Engström (2006)

26. Operational definition Entropyof the prediction errors made by a linear predictive filter applied on the steering wheel angle signal (see Boer 2005 for detailed mathematical definition) Interpretation ”…increase in high frequency steering corrections that result after periods of diverted or reduced attention (i.e., in response to a perceived vehicle drift outside the acceptable tolerance margins that mounted during these periods of degraded information)” (Boer, 2005) Steering entropy (1)

27. Advantages Strongly sensitive to visual and cognitive load in a range of conditions SW data easy to measure, also in the field Relatively robust to differences in driving environment (road type, curvature, test set-up etc.) Disadvantages Fairly complex to compute (though straightforward) Somewhat difficult to interpret, even in terms of performance (increased SEmay indicate both increased and reduced lateral control) Interpretation of free parameters (alpha and re-sampling rate) not entirely straightforward Requires baseline data for computation of task condition data ”Normalisation” to baseline data makes BL and Task data somewhat dependent Relation to crash data No established relation to crash data (only indirectly via visual distraction) Steering Entropy pros and cons

28. Operational definition The number, per minute, of steering wheel reversals larger than a certain angular value referred to as the gap size (see Markkula & Engström, 2006, for detailed mathematical definition) Steering Wheel Reversal Rate (SRR)

29. Representative results from HASTE: SRR1, 1 degree gap size VTEC simulator, rural road Cognitive task Visual task

30. SRR Sensitivity (effect size) as a function of gap size

31. Advantages Strongly sensitive to visual and cognitive load in a range of conditions SW data easy to measure, also in the field Easier to interpret than steering entropy Does not involve normalisation of task data to baseline data (like Steering Entropy) Disadvantages Sensitive to environment factors Somewhat difficult to interpret in terms of performance - increased SRR may indicate both reduced and increased lane keeping performance (however, can be tuned by changing gap-size) Relation to crash data Like other steering wheel metrics, no established relation to crash data (only indirectly via visual distraction) SRR pros and cons

32. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

33. Example data

34. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

35. Total time spent looking away from the road Intensity (”how much looking away per time untit”) Distribution of single glance durations Eccentricity Factors to account for On-road A Off-road B On-road Off-road C On-road Off-road

36. Traditional (ISO 15007) glance-based metrics

37. Automating the ISO 15007 metrics: The VDM Tool (Larsson, 2002; Johansson et al., 2006)

38. Standard originally intended for manual transcription Glance-based metrics are very sensitive to noise Requires careful calibration and signal pre-processing Much data still needs to be discarded (~30% in HASTE) Difficulties with automating the ISO 15007 metrics

39. An alternative: Road centre-based metrics Road Centre On-road glances Off-road glances

40. Operational definition: PRC-Task:The percent of fixations directed towards the road centre (RC) during a task. Represents intensity only. PRC-Window:The percent of fixations directed towards the RC during a moving time window of 1 minute. If the task is shorter than 1 minute, the remaining time is completed with a constant PRC of 80%. The windowing adds a weighting for task duration. Percent Road Centre (Victor, 2005)

41. Example data from HASTE (Victor, Harbluk and Engström, 2005)

42. Advantages Very sensitive to visual task difficulty Allows for baseline data (which glance-based metrics to do not) Should be more robust to measurement noise (focus measurement where eye tracking accuracy is normally best, data order does not matter) Disadvantages PRC-Task measures only intensity PRC-Window accounts for task duration, but somewhat arbitrarily Does not account for eccentricity Relation to crash data Strong empirical evidence on the relation between visual diversion from the forward road scene and accident risk (e.g. Wierwille and Tijerina, 1995; Klauer et al., 2006) Pros and cons of PRC

43. RC-based versions of the ISO metrics (Kronberg et al., 2006) Other ways to account for both intensity and duration Weighting function for single glance duration Account for eccentricity For example: Further ideas gi=single off-road glance duration E=eccentricity weighting function

44. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

45. Operational definition The standard deviation of the combined horizontal and vertical angles. The combined angle is the square root of the sum of squared vertical and squared horizontal angles (Pythagoras theorem) and thus is a one-dimensional angle between the origin and a gaze point Measuring gaze concentration: Standard deviation of gaze angle

46. Effects of cognitive task on gaze concentration Gaze angles (pich and yaw) Baseline Cognitive task (levels 1-3 aggregated) VTEC simulator, rural road

47. Example data from HASTE (Victor, Harbluk and Engström, 2005)

48. Advantages Sensitive to cognitive load (more than PRC) – good metric of gaze concentration Robust to noise since data order does not matter Disadvantages Only applicable to assessment of purely cognitive load Relation to crash data No empirical data on the relation between gaze concentration and crash risk Pros and cons of SD gaze angle

49. Background of research (HASTE and AIDE) Metrics Lane keeping Steering Eye movements Time sharing Gaze concentration Conclusions, lessons learned and topics for further research Outline

50. The metrics addressed here mainly relevant for evaluating visually demanding tasks Lateral control performance metrics somewhat problematic as surrogate safety metrics – no clear link to crash data Direct eye movement metrics seem to be the most promising (though still practical difficulties with data collection and analysis) For cognitive tasks, other metrics are needed to capture the main safety-relevant effects (e.g. detection task metrics such as PDT) Lack of agreed driver model – very little consensus on how to interpret even the most common driving performance metrics Little discussion and emprical work on the link between performance metrics and safety (especially in Europe) Conclusions and lessons learned