using a low cost electroencephalograph for task classification in hci research n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Using a Low-Cost Electroencephalograph for Task Classification in HCI Research PowerPoint Presentation
Download Presentation
Using a Low-Cost Electroencephalograph for Task Classification in HCI Research

Loading in 2 Seconds...

play fullscreen
1 / 71

Using a Low-Cost Electroencephalograph for Task Classification in HCI Research - PowerPoint PPT Presentation


  • 118 Views
  • Uploaded on

Using a Low-Cost Electroencephalograph for Task Classification in HCI Research. Johnny C. Lee Carnegie Mellon University. Desney S. Tan Microsoft Research. UIST 2006, Montreux Switzerland. NY Times Magazine, October 16, 2005. National Geographic, March 2005. Brain-Computer Interfaces (BCI).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Using a Low-Cost Electroencephalograph for Task Classification in HCI Research' - casey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using a low cost electroencephalograph for task classification in hci research

Using a Low-Cost Electroencephalograph for Task Classification in HCI Research

Johnny C. Lee

Carnegie Mellon University

Desney S. Tan

Microsoft Research

UIST 2006, Montreux Switzerland

slide2

NY Times Magazine, October 16, 2005

National Geographic, March 2005

brain computer interfaces bci
Brain-Computer Interfaces (BCI)

A direct technological interface between a brain and computer not requiring any motor output from the user

Example Conferences/Journals with BCI interests:

Neural Information Processing Systems (NIPS)

IEEE Transactions on Biomedical Engineering

IEEE Transactions on Neural Systems and Rehabilitation Engineering

why is this relevant to uist or hci
Why is this relevant to UIST or HCI?

BCI research traditionally focuses on exploratory neuroscience and rehabilitation engineering.

Brain sensing could provide valuable data about:

- engagement

- cognitive work load

- surprise

- satisfaction

- frustration

Potentially helpful to

Context Sensitive or Evaluation Systems

slide5

Values of HCI

Values of BCI

To use any means necessary to demonstrate that brain-computer interaction is possible.

To use reasonable means to achieve a practical benefit to many users.

We’d like to:

It is okay to:

  • use equipment costing $100K to +$1 million USD
  • use highly invasive surgical procedures
  • require hours or days of operant conditioning
  • remove data from poor performing subjects

use fairly affordable and accessible equipment

be safe for repeated and extended use

be usable without requiring significant user training

use data from all subjects to evaluate its performance

VS

brain sensing imaging technologies
Brain Sensing/Imaging Technologies

MRI

CT

ECoG

SPECT

PET

MEG

fMRI

EROS/fNIR

EEG

Currently Impractical

for HCI

- Safe, easy, no medical expertise

eeg electroencephalograph
EEG – Electroencephalograph

the neurophysiological measurement of the electrical activity

of the brain by recording from electrodes placed on the scalp

(skipping the lower level neurophysiology)

  • Measures the voltage difference between two locations on the scalp
  • Only picks up gross, macroscopic, coordinated, and synchronized firing of neurons near the surface of the brain with perpendicular orientation to the scalp. (thus majority of activity is hidden)

Analogous to holding a thermometer up to the side of a PC case

eeg devices
EEG Devices

Manufacturer: BioSemi

Channels: 64-128

Cost: ~$30K USD

Manufacturer: EGI Systems

Channels: 128-512

Cost: $100K-$250K USD

the brainmaster
The Brainmaster
  • Lowest cost FDA approved device
  • Designed for home and small clinical use.
  • Only $1500 USD
  • Specs:
    • 2-channels
    • 8-bit at 4µV resolution
    • 256 samples/sec
  • Has yet to be validated for BCI research work.
  • If it works, it lowers the entry bar for BCI research.
validating the device and ourselves
Validating the Device (and ourselves)

1.Validate the device

Can we get useful data from such a low-end device?

2. Validate ourselves

To explore this space, we must be able to collect our own data.

validating the device and ourselves1
Validating the Device (and ourselves)

Keirn, Z., “A New Mode of Communication Between Man and His Surroundings”, IEEE Transactions on Biomedical Engineering, Vol. 37, No. 12, 1990.

  • Data is available for download
  • Data has not been reproduced in the past 15 years
    • Some computational BCI researchers have just used this data.
    • State of the art does is not a great deal better.
reproducing the keirn data
Reproducing the Keirn Data

We adapted procedure from Keirn to better control potential confounds.

3 tasks:

  • Rest (Baseline): Relaxation and clearing of mind
  • Math: Mental arithmetic, prompted with “7 times 3 8 5”
  • Rotation: Mentally rotate an object, prompted with “peacock”

Tasks from the original paper were designed to elicit hemispheric differences.

experimental procedure
Experimental Procedure

User is instructed to keep eyes closed, minimize body movement, and not to vocalize part of the tasks.

For each task, a computer driven cue is given:

Rest, Math, Rotate

Following Math and Rotate, the experimenter says either the math problem or object

experimental procedure1
Experimental Procedure

Block design adapted from Kiern

task (14 seconds)

trial

Rot

Math

Rest

Math

Rest

Rot

Rest

Rot

Math

Rot

Rest

Math

Rest

Math

Rot

Math

Rot

Rest

session

experimental procedure2
Experimental Procedure

Rot

Math

Rest

Rot

Math

Rest

Rot

Math

Rest

Math

Rest

Rot

Math

Rest

Rot

Math

Rest

Rot

Rest

Rot

Math

Rest

Rot

Math

Rest

Rot

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Rest

Math

Rest

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Rest

3 sessions per subject

Many short tasks prevent correlation with EEG drift

experimental procedure3
Experimental Procedure

Rot

Math

Rest

Rot

Math

Rest

Rot

Math

Rest

Math

Rest

Rot

Math

Rest

Rot

Math

Rest

Rot

Rest

Rot

Math

Rest

Rot

Math

Rest

Rot

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Rest

Math

Rest

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Math

Rot

Rest

Math

Rot

Rest

Math

Rot

Rest

Subjects:

8 subjects (3 female)

29-58 years of age

All were cognitively and neurologically healthy

All right handed

eeg setup
EEG Setup

International 10-20 EEG electrode placement system

Two channels placed on P3 and P4 with both references tied to Cz.

Electrodes are held in place using conductive paste.

5-10 minute preparation.

data processing
Data Processing

task (14 seconds)

Rot

14 secs

data processing1
Data Processing

task (14 seconds)

Rot

14 secs

Task Cue

data processing2
Data Processing

task (14 seconds)

Rot

14 secs

Experimenter Prompt

data processing3
Data Processing

task (14 seconds)

Rot

14 secs

Task Onset

data processing4
Data Processing

task (14 seconds)

Rot

14 secs

Performing Task

data processing5
Data Processing

task (14 seconds)

Rot

14 secs

~4 secs

Performing Task

data processing6
Data Processing

task (14 seconds)

Rot

10 secs

Performing Task

removing time for machine learning
Removing time for machine learning

Most machine learning algorithms don’t handle time series data very well.

10 seconds

removing time for machine learning1
Removing time for machine learning
  • Divide the 10 seconds into 2 sec windows that overlap by 1 sec
  • Perform signal processing on each of the 9 windows to get our “time less” feature set

2 secs

removing time for machine learning2
Removing time for machine learning
  • Divide the 10 seconds into 2 sec windows that overlap by 1 sec
  • Perform signal processing on each of the 9 windows to get our “time less” feature set

2 secs

removing time for machine learning3
Removing time for machine learning
  • Divide the 10 seconds into 2 sec windows that overlap by 1 sec
  • Perform signal processing on each of the 9 windows to get our “time less” feature set

2 secs

removing time for machine learning4
Removing time for machine learning
  • Divide the 10 seconds into 2 sec windows that overlap by 1 sec
  • Perform signal processing on each of the 9 windows to get our “time less” feature set

2 secs

removing time for machine learning5
Removing time for machine learning
  • Divide the 10 seconds into 2 sec windows that overlap by 1 sec
  • Perform signal processing on each of the 9 windows to get our “time less” feature set

This provides

486 windows

per participant

2 secs

signal features for each window
Signal features for each window

Generic signal features such as mean power, peak frequency, peak frequency amplitude, etc.

Features frequently used in EEG signal analysis.

common eeg features
Common EEG Features

Raw EEG

Spectral

Power

Theta

(4Hz-8Hz)

Beta Low

(12Hz-20Hz)

Beta High

(20Hz-30Hz)

Gamma

(30Hz-50Hz)

Delta

(1Hz-4Hz)

Alpha

(8Hz-12Hz)

feature processing and selection
Feature Processing and Selection

The 39 base features from each window are mathematically combined to create 1521 total features.

We used a feature preparation and selection process similar to [Fogarty CHI’05] to reduce the number of features:

23 features for 3-task classification (486 examples)

16.4 features for pair-wise classification (324 examples)

baseline results 3 cognitive tasks
Baseline Results – 3 cognitive tasks

BayesNet classifier

Chance:33.3% 50% 50% 50%

slide37

2 secs

86.5%

68.3%

82.9%

83.8%

throwing time back in
Throwing time back in…

We can average over temporally adjacent windows to improve classification accuracy

“Math”

averaging with task transitions
Averaging with Task Transitions

Task transitions result in conflicting data in averaging window.

High density of transitions will result in lower accuracy.

averaging with task transitions1
Averaging with Task Transitions

Fewer task transitions will yield better classification accuracy.

averaging with task transitions2
Averaging with Task Transitions

No transitions and averaging over all data will be the even better.

classification accuracy with averaging
Classification Accuracy with Averaging

+5.1 to +15.7%

for 3-tasks

Error bars represent standard deviation

so can we really read minds
So, can we really read minds?

Possibly not, we might be really detecting subtle motor movements….

Error bars represent standard deviation

cognitive motor fabric
Cognitive/Motor “Fabric”

Tasks of varying cognitive difficultly are involuntarily coupled with physiological responses, such as minute imperceptible motor activity. [Kramer ’91]

Therefore, it is impossible to completely isolate cognitive activity neurologically intact individuals.

Does this matter to neuroscience? Yes

Does this matter to HCI? Maybe not

cognitive motor fabric1
Cognitive/Motor “Fabric”

If motor artifacts are reliably correlated with different types of tasks or engagement, why not use those to help the classifier?

Requiring users to not move is also very impractical.

Non-Cognitive Artifacts detected by EEG:

  • Blinking
  • Eye movement
  • Head movement
  • Scalpal GSR
  • Jaw and facial EMG
  • Gross limb movements
  • Sensory Response Potentials
experiment 2 game task
Experiment 2 – Game Task

To explore this idea of using non-cognitive artifacts to classify tasks using EEG, we chose a PC-based video game task.

Halo, a PC-based first person shooter game produced by Microsoft Game Studios.

Navigate a 3D environment in an effort to shoot opponents using various weapons.

Relatively high degree of interaction with mouse and keyboard input controls.

game tasks
Game Tasks
  • Rest – baseline rest task, relax, fixate eyes on cross hairs on center of screen, do not interact with controls. Game elements do not interact with participant.
  • Solo – navigate environment, interact with elements in the scene, and collect ammunition. Opponent controlled by expert did not interact with participant.
  • Play – navigate environment and engage opponent controlled by expert. Expert instructed to play at a level just slightly above skill of participant to optimally challenge them.
game experimental procedure
Game Experimental Procedure
  • Setup, design, and procedure was similar to first study.
  • Participants had tutorial and practice time with game controls.
  • 3 tasks repeated 6 times (counterbalanced)
  • Tasks were 24 seconds to allow navigation time.
  • Only 2 sessions were run for each participant
  • Same 8 participants from first study were run in this study.
  • Same data preparation and machine learning procedure.
results game tasks
Results – Game Tasks

93.1%

Error bars represent standard deviation

conclusion
Conclusion

This experimental design and data processing procedure can be applied to a much wider range of applications/tasks. Our two experiments were just two examples at different ends of a spectrum.

Compelling results can be achieved with low-cost equipment and without significant medical expertise or training.

Non-cognitive artifacts (inevitable in realistic computing scenarios) can be embraced improve classification power.

To make BCI relevant to HCI, we must challenge traditional assumptions and creatively work with its limitations.

thanks
Thanks!

Johnny Chung Lee

johnny@cs.cmu.edu

Desney Tan

desney@microsoft.com

Thanks to MSR and the VIBE

Group for supporting this work.

brain sensing imaging technologies1
Brain Sensing/Imaging Technologies

MRI – only anatomical data

CT – only anatomical data

ECoG

SPECT

PET

MEG

fMRI

EROS/fNIR

EEG

brain sensing imaging technologies2
Brain Sensing/Imaging Technologies

MRI – only anatomical data

CT – only anatomical data

ECoG – highly invasive surgery

SPECT – radiation exposure

PET – radiation exposure

MEG

fMRI

EROS/fNIR

EEG

brain sensing imaging technologies3
Brain Sensing/Imaging Technologies

MRI – only anatomical data

CT – only anatomical data

ECoG – highly invasive surgery

SPECT – radiation exposure

PET – radiation exposure

MEG – extremely expensive

fMRI – extremely expensive

EROS/fNIR

EEG

brain sensing imaging technologies4
Brain Sensing/Imaging Technologies

MRI – only anatomical data

CT – only anatomical data

ECoG – highly invasive surgery

SPECT – radiation exposure

PET – radiation exposure

MEG – extremely expensive

fMRI – extremely expensive

EROS/fNIR – currently expensive, still in infancy

EEG – safe, easy, no medical expertise

event related potentials erp
Event Related Potentials (ERP)
  • Electrical activity related to or in response to the presentation of a stimulus
  • Very well studied
  • Relatively robust
  • Used daily in clinical settings to check sensory mechanisms, typically in infants
  • Requires averaging over 30-100 windows synchronized with to see response.
erp aep
ERP - AEP

ERP: Auditory Evoked Potential

  • Used in clinics/hospitals to check hearing.
  • Response to clicks in the ear

AEP response

Bold Lines = no clicks

Thin Line = with clicks

AEP response

erp vep
ERP - VEP

ERP: Visual Evoked Potential

  • Focusing on a flashing target, the visual cortex will “resonate” at the stimulus frequency.

Stimulus Frequency

Harmonics

erp auditory and visual p300
ERP – Auditory and Visual P300
  • Well known/studied potential related to “attention” or “surprise”
  • Presented with 2 stimuli and instructed to count one of the stimuli
  • Positive response will follow the stimulus of interest.
side note eeg as ecg
Side note: EEG as ECG

ECG - Electrocardiogram

  • placing an electrode on the chest provides a clear measure of cardiac activity.
  • translation to BPM is a simple autocorrelation

0.4 µV units

Single Beat period

Heart beats

eeg as emg
EEG as EMG

EMG - Electromyography

  • Measures muscular activity

Wrist relaxation (return to straight position)

Wrist rest state

0.4 µV units

Tension holding

Wrist inward contraction (toward inner forearm)

NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement.

eeg as blink detector
EEG as Blink Detector
  • Electrical activity due to muscle movement involved with eye blinks propagate through the head.
  • Similarly, eye movements also affect the EEG recording

Blinks

Blinks

task classification background
Task Classification Background
  • Previous work is split primarily into two camps:

Operant Conditioning

Pattern Recognition

task classification background1
Task Classification Background
  • Previous work is split primarily into two camps:

Operant Conditioning

Pattern Recognition

Human learns how

the machine works

Machine learns how

the human works

task classification background2
Task Classification Background
  • Previous work is split primarily into two camps:

Operant Conditioning

Pattern Recognition

Human learns how

the machine works

Machine learns how

the human works

Relatively new

Early dabbling in the late-80’s

Most work has happened in last 5 years.

pattern recognition
Pattern Recognition

Data Collection &

Experimental Design

Signal Processing &

Feature Generation

Machine Learning &

Improving Accuracy

eeg setup1
EEG Setup

International 10-20 EEG electrode placement system

Two channels placed on P3 and P4 with both references tied to Cz. Locations selected based on pilot recordings.

Attaching electrodes: Prepare the site with a cleaner, use conductive paste to improve connection and hold electrode in place.

P Paste rinses out with water, non-toxic. 5-10 minute preparation.