Software Tools for Visualization of Multimodality Brain Data for Language Mapping

1. Software Tools for Visualization of Multimodality Brain Data for Language Mapping 428.16 Andrew V. Poliakov, PhD1, Eider B. Moore1, David P. Corina, PhD2, James F. Brinkley, MD, PhD1 1Structural Informatics Group, Department of Biological Structure, 2Department of Psychology, University of Washington, Seattle, Washington USA Abstract Our group has been developing software tools for processing, integrating and visualizing multimodality data for language mapping. The primary data are collected in preparation for and during neurosurgery for intractable epilepsy, and include structural and functional MRI and surgical photographs of cortical stimulation maps (CSM). Software developed by our group provides tools for reconstructing the cortical surface, veins and arteries; mapping CSM sites onto the reconstructed surface; visualization and analysis, including web-based visualization. These tools, however, operate only on Unix platforms (SGI Irix and Linux), and are non-portable and difficult to maintain. We present BrainJ3D -- a new software toolkit for processing and visualizing language mapping and other brain mapping data. BrainJ3D is developed in Java/Java3D, and is cross-platform (Windows, Linux and Solaris). The toolkit supports mapping of CSM sites onto the cortical surface, and editing and labeling such maps. It provides general purpose visualization tools, including image and slice viewers, interactive visualization of 3D scenes showing cortex, veins, arteries and other surfaces, and cut-away views of the brain. Functional data (e.g. fMRI, ERP, spectroscopy, etc.) can be overlaid onto the structural volumes and surfaces. The data are organized in a coherent and flexible workspace. BrainJ3D utilizes hardware-accelerated surface rendering (3D graphics card and Java/Java3D are required). In addition to standalone mode, the program can operate in a client/server mode, allowing remote visualization of the data stored on a remote server, which performs the rendering and returns them as 2-D images to a simple client program on a local computer. Such capabilities will permit the development of spatial query engines and remote visualization of results obtained from distributed brain map databases. Support: Human Brain Project grant DC02310. • Web-based Information System for Language Mapping • The experiment management component of our information system allows users to store numeric and text-based data (e.g. patient demographics, transcripts of experimental trials, etc.) in a relational database, and edit and manage these data on the web2. The experiment management component is also used to organize 2-D images (intra-operative photographs etc.). • The 3-D visualization component is used to visualize 3-D volumes (e.g. structural and functional MRI.), 3-D surfaces derived from such volumes (e.g. cortex, veins and arteries) and other 3-D data (CSM maps). These data are loaded into a graphics server, rendered, then saved as 2-D snapshots, which are sent to the client side. The key features of this approach include: • Client-server model • Server-based data storage and rendering • Simple web-based clients (e.g. Java or cgi-based) BrainJ3D BrainJ3D is our next generation tool for visualizing brain mapping data. It builds on our previous experience with 3-D visualization of language mapping data. It takes advantage of a modern programming environment provided by Java and Java3D. It is designed to allow us to support the same functionality yet overcome certain limitations of our current system. BrainJ3D is designed to be: • Flexible, making it easy to implement new features and functionality • Cross-platform, since it is written in Java/Java3D • Interoperable with several popular brain imaging software packages Experiment Management Component Background As part of the Human Brain Project we are developing software tools for processing, integrating and visualizing multimodality data for language mapping. Prior to surgery for intractable epilepsy, structural (MRI, MRV, MRA) and functional magnetic resonance (fMRI) images are collected. During surgery Cortical Stimulation Mapping (CSM) is used to locate areas of language on the cortex that need to be avoided1. These cases present a unique opportunity for understanding language function because the CSM data provide a “gold standard” against which fMRI and other non-invasive methods can be compared. To process these data we are developing a web-based information system for managing and analyzing language mapping data, which includes an experiment management component as well as a 3-D visualization component. 3-D Visualization Component • Current work • Integration with Experiment Management component • Beta-release in the near future • Will require extensive testing as well as improving usability and refining graphics user interface. • Improve interoperability • Interoperability proved to be a fast moving target, and is often beyond our control • Support for other data modalities • e.g. Spectroscopy, ERP source signal, Transcranial Magnetic Stimulation • Generalization to other applications • Teleradiology • Treatment planning • Experiment management • Online medical record Client #1 CGI script Client #2 Java Applet Client #3 . . . Web Interface Client Internet Graphics Server Server snapshot snapshot Relational Database Features of BrainJ3D • Cut-away mode with functional data overlay • Mapping and labelling 3D points on the cortex surface surfaces, e.g. CSM • Surface-based 3D visualization hardware accelerated, high performance Slice viewer with functional overlay • Data is organized in a customisable workspace tree • Supports popular file types from other software packages (SPM, FSL etc.) • Standalone and client/server mode It leverages Java RMI (Remote Method Invocation) to support client/server mode. The server uses off-screen rendering to generate 2D snapshots that are then sent to the client. • Open architecture and file formats XML files used for all internal configurations and settings Data 3-D Image Volumes 3-D Models Stimulation Sites 2-D Images • Motivation for a Next-generation 3-D Visualization Component • Ourcurrent production visualization techniques are based on Skandha4, an in-house general purpose graphics toolkit3. Using this toolkit, we were able to solve a number of problems, including reconstruction of the cortical surface, veins and arteries, mapping language data onto these 3-D models, integrating other data modalities such as functional MRI, and web-based visualization4. However, as computer programming and graphics continue to change and evolve, we face several challenges when maintaining and further developing these 3-D applications. These challenges include: • Relatively slow software rendering, • Non-portable architecture (SGI Irix and Linux only), • Large codebase that needs to be maintained and developed, • Reliance on software tools that are becoming outdated or obsolete • Because of these considerations, we decided to explore alternate modern techniques. BrainJ3D is the result of these explorations. Acknowledgements This work was funded by Human Brain Project grant DC02310, National Institute of Deafness and Other Communication Disorders and National Institute for Mental Health. Why Java • Modern object-oriented language • Numerous libraries and utilities • Web-based and remote technologies are well supported. Why Java3D Java3D is a high level 3-D programming language that provides several advantages over lower level approaches such as OpenGL. • Cross Platform ( Linux, Windows and Solaris) • Supports hardware acceleration and scene optimisation, so it performs as well as programs written in OpenGL. • Supports hardware accelerated off-screen rendering. • Mature software package that provides needed functionality • References • Ojemann GA. Mapping of neuropsychological language parameters at surgery. Int Anesthesiol Clin 1986 Fall;24(3):115-31 • R. M. Jakobovits and J. F. Brinkley, Managing medical research data with a Web-interfacing repository manager, Proceedings, AMIA Fall Symposium, Nashville, pp. 454-458, 1997. • Modayur BR, Prothero J, Ojemann, G, Maravilla K, Brinkley JF. Visualization-based mapping of language function in the brain. Neuroimage, 1997: 6: 245-258. • A. V. Poliakov, K. P. Hinshaw, C. Rosse and J. F. Brinkley, Integration and Visualization of Multimodality Brain Data for Language Mapping, Proceedings, AMIA Fall Symposium Washington, D.C., pp. 349-53, 1999.