Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status

1. PLS Tools in ElectroencephalographyLeonard J. TrejoPDT InstitutePalo Alto, CA 94303, USAThe 6th International Conference on Partial Least Squares and Related MethodsSept. 4th – 7th, 2009Beijing, China • Sponsored by • US Army Research Office STIR: Advanced Estimation of Cognitive Status • Contract No. W911NF-08-C-0121 15-SEP-2008 TO14-MAR-2009

2. PDT and PDT Institute • PDT • Neuroergonomics Models and Applications • Human-System Integration • Human Performance Optimization • Robust Biomedical Signal Processing • Embedded and Real-time Systems for Bio-Sensing • PDT Institute • PhD/Masters/Undergraduate Training • University Partners (UC Santa Cruz, Tsinghua University, UC San Diego, Univ. of West Florida)

3. When I am not working…

4. Outline • Problem: stress, workload, fatigue and performance • Response: Neuroergonomic models and control systems • Create useful definitions of cognitive states • Model, estimate and control cognitive states • Background • Multimodal sensor-state models using PLS and KPLS Algorithms • Successes and failures: fatigue / BCI / engagement / workload • New directions • Truly multidimensional sensor-process models • PARAFAC, N-PLS • Summary

5. Estimation of Cognitive States Biosignals Models and Algorithms

6. Useful Definitions • Engagement: selection of a task as the focus of attention and effort • Workload: significant commitment of attention and effort to an engaged task • Visual, Auditory, Haptic • Psychomotor • Cognitive (memory, executive) • Overload: task demands outstrip performance capacity • Mental Fatigue: desire to withdraw attention and effort from an engaged task associated with extended performance (~45 min)

7. Electroencaphalogram • Cerebral Cortex • the outermost layers of brain • 2-4 mm thick (human)

8. EEG Sources A pyramidal neuron with a soma, apical & basal dendrites and a single axon

9. EEG Sources A pyramidal neuron with a soma, apical & basal dendrites and a single axon

10. Other Elements of Sensor-State Models

11. Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued Parietal Fz Pz Alpha Frontal Theta

12. Robust EEG-Based Classification of Mental Fatigue 2300 (Day 1) vs. 1900 Hrs (Day 2) 100 90 80 Test Proportion Correct 70 60 50 40 -18 -15 -12 -9 -6 -3 0 50 56 79 94 99 100 100 21 Channels 50 51 65 88 96 98 98 12 Channels 50 54 76 88 90 88 87 4 Channels Signal-to-noise Ratio (dB)

13. Successful Application 2: BCI

14. Successful Application 2: BCI

15. Stress, Workload, Fatigue and Performance Trejo, et al. ACI 2007

16. Cognitive Overload (Trejo, et al. ACI 2007) Trejo, et al. ACI 2007 Trejo, et al. ACI 2007

17. Stabilizing Classifiers

18. Multimodal Overload Patterns Value Time (s)

19. Workload-related EEG Sources

20. Application Summary • Individual models of engagement and fatigue: • 90-100% accurate • Stable within a day • Stable from day to day • Individual models of workload or effort: • 60-90% accurate • Moderately stable within a day • Unstable from day to day • Normative models (limited data): • 50-70% accurate • Moderately stable

21. Recommended Directions • Deployable multimodal sensors (EEG, fNIR, EOG, gaze, HRV, EMG, SCR, SpO2,BP, core body temperature, gesture, posture facial expression, ...) • Multimodalexperimental designsandoperational tests • Advanced neurocognitive process models • Multimodal sensor-processmapping algorithms APECS

22. “Atomic” Decomposition • “In the parlance of modern harmonic analysis (Chen and Donoho, 2001), we performed a space/ frequency/time ‘‘atomic decomposition’’ of multidimensional data. In other words, we assume that each neural mass contributes a distinctive atom to the topographic frequency/time description of the EEG, so that the estimation of these atoms is possible by means of signal-processing techniques. Each atom will be defined by its topography, spectral content, and time profile; in other words, by its spatial, spectral, and temporal signatures.” • Fumikazu Miwakeichi, et al, Decomposing EEG data into space–time–frequency components using Parallel Factor AnalysisNeuroImage 22 (2004) 1035–1045. • Chen, S., Donoho, D., 2001. Atomic decomposition by basis pursuit. SIAM Rev. 43, 129– 159.

23. “Atomic” EEG Elements Basic Sources“atoms” CoherentSystems “molecules” Atoms Molecule Covalent Bonds Coherence Bonds

24. “Molecular” EEG Processes Coherence Bonds Atoms

25. bf af Familiar (bilinear) Mapping Algorithms Factor Analysis Principal Component Analysis (PCA)

26. Multimodal Mapping • How to generalize bilinear models to systems with more dimensions? • Unfolding a bilinear model • Represent all experimental factors in one dimension • Observations (trials) is second dimension • Contrast each dimension vs. pairs of the other two • Multidimensional model • Assume orthogonal factors: PARAFAC • Allow interacting factors: Tucker 3 • Modeling approach • Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3 • Supervised extraction: N-PLS

27. X X1 X2 X3 Dim 2 & 3 Dim 1 & 3 Dim 1 & 2 Dim 2 Dim 3 Dim 1 Unfolding a Bilinear Model X Unfolding

28. Multidimensional Modeling(Tucker 3 Model, unsupervised) • xijkis an element of (l x m x n)multidimensional array • F1, F2, F3are the number of components extracted on the 1st, 2nd and 3rd mode • a, b, c are elements of the A, B, C loadings matrices for the 1st, 2nd and 3rd mode • gare the elements of the core matrix G which defines how individualloading vectors in different modes interact • eijkis an error element (unexplained variance)

29. PARAFAC (Parallel Factor Analysis, unsupervised) PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array: bf cf af

30. electrodes af – spectral atom bf – spatial atom cf – temporal atom EEG time frequency max. covariance vf – workload atom uf – temporal atom X time Labels workload condition N-way PLS(supervised) X cf uf bf af vf

31. Demo: Workload / PARAFAC EEG Workload condtions (e.g., trials, time) Electrodes EEG Frequency

32. Summary • Successes and Failures • Individual models of engagement and fatigue: accurate, stable • Individual models of workload or effort: variably accurate, unstable • Normative models (limited data): inaccurate, unstable • Useful models of state-related EEG sources • “Atomic” EEG sources • “Molecular” EEG systems • Approaches to multidimensional models and algorithms • Tradtional bilinear methods (PCA, factor analysis, ICA) • Truly multidimensional methods • Correlated factors (Tucker 3) • Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS) • Supervised algorithms (N-PLS)

33. Within-day Results