Chapter 23 Combining BCI and Virtual Reality : Scouting Virtual Worlds

1. Chapter 23Combining BCI and Virtual Reality: Scouting Virtual Worlds

2. Introduction • Before a BCI can be used for control purposes, several training sessions are necessary • Operant conditioning • Feed back, real-time changes to the user • Machine learning • Adaptive algorithms to detect brain patterns

3. VR (Virtual Reality) • A technology which allows a user to interact with a computer-simulated environment.

4. VR (Virtual Reality) • VE (Virtual Environment) • Allow users • to be shielded from the outside world • to be able to focus on the required mental task

5. Then, Why VR for BCI? Feedback • Visualization vs. VE feedback • Ron Angevin et al. (2004) • Control group (standard BCI feedback) reacted faster • VR group achieved less error

6. Graz-BCI • Basic principle • detection & classification of motor-imagery related EEG patterns • Sensorimotor rhythms are analyzed • C3, Cz and Cz

7. Graz-BCI + VE • VE • To let a user become immersed in a 3D scene • Cave • Multiprojection stereo-based head-tracked VE system

8. Graz-BCI + VE • 3D virtual environment • Creation of a 3D model of the scene • 3D modeling Software Packages Performer • Maya • Generation of a VR-application that controls and animates the modeled scene • Virtual Research V8 HMD • 640 X 480 pixels, refresh rate 60Hz • Vrjuggler + single back-projected wall + shutter glasses • Cave-like system

9. Graz-BCI + VE • BCI experiments • Require a subject in a sitting position • No positional information had to be considered • Rotational information from the tracking system was ignored • Rotation should be controlled by the BCI

10. Study 1 Roation in a VE by Left- and Right-Hand Motor Imagery

11. Experiment • Imagination of left and right hand movement • Subjects • 2 male(23, 26 years old), 1 female(28 years old) • 7 months • Feedback conditions • A standard horizontal bar graph on a desktop monitor • Virtual conference room presented with an HMD • Virtual pub populated with animated avatars (including music and chatter of the avatars)

12. Experiment • The order of feedback conditions • Bar graph → HMD → Cave → HMD → bar graph • Instruction • To imagine left or right hand movements depending on an acoustic cue (single or double beep) • Control • Either the length and the orientation of the horizontal bar graph (in case of the standard BCI feedback) • Rotation angle and direction within VR

13. Experiment • During experiments • Cue at second 3 • Feedback for 4s • Screen update (including rotation) 24 times/s • One run 40 trials

14. Results

15. Results • No difference between HMD and Cave • Performed well with VR than bar graph

16. Study 2 Moving Forward in a Virtual Street by Foot Mortor Imagery

17. Experiment • Imagination in the virtual street • Right hand movement: to stop • Foot movement to move (constant speed) • Walking distance is scored • CAM (Cumulative achieved mileage) • Male 23, 28 and 30 years old.

18. Results

19. Study 3, 4 Scouting through a Virtual Apartment Asynchronous freeSpace Experiments

20. Experiment • Virtual apartment on a single back-projected stereoscopic wall • Subject could decide freely where to go along predefined pathways (through the corridors or rooms) • Turn right, left, or straight) • System automatically guided the subject to the next junction • Small map in the bottom right corner of the display

21. Results

22. Conclusion • Using VR • High classification accuracy (low error rate) can be achieved. • Subjects felt more natural in VE compared with BCI experiments with standard feedback • Each subject preferred the Cave experiments to the HMD and both were favored over BCI session on a desktop PC • Motivation seems to improve the BCI performance, but too much excitement might also distract the subject • Despite distraction from auditory and moving visual stimuli in VE, motor imagery and its classification in the ongoing EEG is still possible

23. Thank you !