Interactive exploration in virtual environments
This presentation is the property of its rightful owner.
Sponsored Links
1 / 61

Interactive Exploration in Virtual Environments PowerPoint PPT Presentation


  • 114 Views
  • Uploaded on
  • Presentation posted in: General

Interactive Exploration in Virtual Environments. Robert G. Belleman Universiteit van Amsterdam Email: [email protected] Overview. Large data spaces Interactive exploration environments Objectives Design An example: Simulated vascular reconstruction in a virtual environment

Download Presentation

Interactive Exploration in Virtual Environments

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Interactive exploration in virtual environments

Interactive Exploration in Virtual Environments

Robert G. Belleman

Universiteit van Amsterdam

Email: [email protected]


Overview

Overview

  • Large data spaces

  • Interactive exploration environments

    • Objectives

    • Design

  • An example:

    • Simulated vascular reconstruction in a virtual environment

  • MSc projects


Data explosion

Data explosion

  • High Performance Computing:

    • computing power increases (Moore’s “Law”)

    • storage capacity increases

      Result: “data spaces” get increasingly largeand complex; multi-dimensional, time dependent.


What are data spaces

What are “data spaces”?

Roughly speaking:

  • Data sets

    • the “files” or “states” that are generated

  • Parameter spaces

    • the number of “variables” and their allowed “freedom” (range, resolution) in a program


Large data sets examples

Large data sets: examples

  • Medical images (i.e. CT, (f)MRI, PET)

    • 512 x 512 at 16 bits slices in common use

    • tens to hundreds slices: 5Mb to5Gb per scan

    • 1024 x 1024 at 16 bit in verynear future: 20Mb to 20Gbper scan

    • time variant scans


More examples

… more examples ...

  • Simulation experiment results

    • FEM, MD, lattice Boltzmann

    • often dimensionality > 3(e.g. time variant)

    • multiparameter data fields

    • gigabytes of data per “run”


More examples1

… more examples ...

  • Measurements

    • high-speed data acquisition devices: particle accelerators (Large Hadron Collider, LHC), microbeam scanners, DNA scanners, CLSM

    • Financial data, etc.

      Terabytes of data per experiment is no longer an exception!


Parameter spaces

Parameter spaces

  • Simulation of complex systems

    • intractable: a certain timestep in a simulation can only be reached by starting at t=0

    • NP complete: time and space requirements grow exponentially with problem size

    • Explicit simulation by a guided search through parameter space required (non-deterministic algorithms: SA, CA, NN, LBM, etc.)


Parameter spaces examples

Parameter spaces: examples

  • Molecular dynamics

    • picosecond timeresolution

    • Molecular docking involves searchthrough large problemspaces


More examples2

… more examples …

  • Finite Element Methods (FEM)

    • large (hierarchically) structured meshes

  • Lattice Boltzmann methods (LBM)

    • large 3D (hierarchically) structured grids

    • large parameter spaces

  • “Optimization” problems in general


From data to knowledge

From data to knowledge

  • Analysis of data spaces is often difficult

    • no analysis methods known, or ill-posed

    • size of data sets too large or too complex

      Often data analysis or simulation runs can take days, sometimes weeks!


Bring in the expert

Bring in the expert

Presentation is often the only way to obtain insight (note: not limited to visualization)

Is it possible to make short cuts? E.g. by putting “an expert” in the loop?


Human in the loop

Human In The Loop

  • HITL

    • a.k.a. “interactive exploration”

    • a.k.a. “exploratory analysis”

    • a.k.a. “computational steering”

    • a.k.a. “problem solving environments”

    • a.k.a. “virtual laboratory”


Interactive exploration environments

Interactive Exploration Environments

Objective: to provide an interactive environment that allows for the exploration of large data spaces.

Responsive, fast, accurate, reliable, easy-to-use, scalable, efficient, versatile, etc.

The design of Interactive Exploration Environments is very multidisciplinary.


Prerequisites

Prerequisites

  • Intuitive interaction

    • increased functionality requires a well considered user interface

  • Real-time feedback: < 0.1 sec delay

  • Quality presentation

    • Informative, avoid clutter

    • Why Virtual Reality?

    • Rapid update rate: for continuous perception:

      • > 10 fps for vision

      • > 20 cps for sound

      • > 1000 cps for haptics

        These often conflict one another.


Design

Design

  • MVC (Model View Controller)design pattern

Model

Data values

Computation

View

Translators

Visualization

State request

Representation

State

Update state

Update view

Controller

Session control

Interaction

User input

User


Interactive dynamic exploration environments idee

Interactive Dynamic Exploration Environments (IDEE)

  • Exploring dynamically changing data from “living” simulations

    • Changing parameters: What if...?

  • Requires “coordination”:


Coordination

Coordination

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

visdata = visualize(simdata, params);

render(visdata, params);

params = interaction();

}

Synchronous

(lockstep)


Coordination1

Coordination

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

}

params = Initialize();

running = true;

While (running) {

visdata = visualize(simdata, params);

}

params = Initialize();

running = true;

While (running) {

params = interaction();

}

params = Initialize();

running = true;

While (running) {

render(visdata, params);

}

Asynchronous


Coordination2

Coordination

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

}

params = Initialize();

running = true;

While (running) {

visdata = visualize(simdata, params);

}

params = Initialize();

running = true;

While (running) {

params = interaction();

}

params = Initialize();

running = true;

While (running) {

render(visdata, params);

}


Coordination on a single host

Shared memory

Coordination on a single host

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

}

params = Initialize();

running = true;

While (running) {

visdata = visualize(simdata, params);

}

params

simdata

Question: What is wrongwith this method?

visdata

params = Initialize();

running = true;

While (running) {

params = interaction();

}

params = Initialize();

running = true;

While (running) {

render(visdata, params);

}


Coordination on a single host1

Coordination on a single host

params = Initialize();

running = true;

While (running) {

P(simdata_sema, params_sema);

simdata = compute(params);

V(simdata_sema, params_sema);

}

params = Initialize();

running = true;

While (running) {

P(simdata_sema, visdata_sema, params_sema);

visdata = visualize(simdata, params);

V(simdata_sema, visdata_sema, params_sema);

}

Shared memory

params

semaphores

simdata

visdata

params = Initialize();

running = true;

While (running) {

P(params_sema);

params = interaction();

V(params_sema);

}

params = Initialize();

running = true;

While (running) {

P(visdata_sema, params_sema);

render(visdata, params);

V(visdata_sema, params_sema);

}


Coordination on distributed hosts

Coordination on distributed hosts

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

}

params = Initialize();

running = true;

While (running) {

visdata = visualize(simdata, params);

}

Computing Cluster

Vector processor

Peer-to-peer networkconnections

But communicationtakes time (precioustime)…

PDA

CAVE

params = Initialize();

running = true;

While (running) {

params = interaction();

}

params = Initialize();

running = true;

While (running) {

render(visdata, params);

}


Communication latency hiding

Communication latency hiding

  • Multiple connections

    • Waiting for acknowledgements is hidden

    • Packets can travel through different routes


Payload reduction

Payload reduction

  • Data encoding

    • Decreases level of detail

      • Accuracy determined by type of representation

    • Lossy compression

    • Induces latency

  • “Standard” compression libraries (zlib)

    • Reduction to 10% not uncommon

    • Lossless compression

    • Induces latency


Communication pipeline

Communication pipeline

  • Cascade of encoding, compression and multiple socket connections


Pipeline performance

Pipeline performance

Latency hiding:

Payload reduction:


Coordination on distributed hosts1

Tuple space

Coordination on distributed hosts

params = Initialize();

running = true;

While (running) {

simdata = compute(params);

out(simdata, timestep);

}

params = Initialize();

running = true;

While (running) {

in(simdata, timestep);

visdata = visualize(simdata, params);

out(visdata, timestep);

}

Computing Cluster

Vector processor

{simdata, 2}

Operations:

out insert

rd blocking read

rdp non-blocking read

in blocking read and delete

inp non-blocking read and delete

{visdata, 2}

{simdata, 3}

{params}

PDA

CAVE

params = Initialize();

running = true;

While (running) {

params = interaction();

out(params);

}

params = Initialize();

running = true;

While (running) {

inp(visdata, timestep);

render(visdata, params);

}


Time management

Rollback

Time management

  • Continuous time

  • Discrete time

  • Discrete event

    • Conservative

    • Optimistic -> rollback

      High Level Architecture (HLA, DMSO):

  • Federates in a federation exchange time-stamped messages (sounds familiar?)

  • Federates may be a combination of:

    • Time regulating or not

      • these advance time at will

    • Time constrained or not

      • these request time advance and wait for grant

  • TSO: Time Stamped Order

    • Messages may not be timestamped in the past

!

!

!


Vr interaction techniques

VR interaction techniques

  • XiVE: X in Virtual Environments

  • CSSR: Context Sensitive Speech Recognition

  • GEOPROVE: Geometric Probes for VEs

  • SCAVI: Speech, CAVE and VTK Interaction


Vr interaction techniques1

VR interaction techniques

XiVE: X in Virtual Environments

  • There is no standard WIMP toolkit for VEs

  • XiVE “swallows” GUIs into a VE

Allows existing applications to be used in VEs without change.


Vr interaction techniques2

VR interaction techniques

CSSR: Context Sensitive Speech Recognition

  • Interaction with visual constructs can be hard in a VE

  • Speech is a different modality

  • Adding context decreases word-error-rate

  • Fast, intuitive interaction

    • “come here”, “make blue”, “increase size by 200%”


Vr interaction techniques3

VR interaction techniques

SCAVI: Speech, CAVE and Vtk Interaction

  • Direct interaction with Vtk “actors” using pointer or voice

    • select, drag, scale, rotate, copy, paste, etc.

    • event handlers; whenin focus, when dragged,when selected, whenspoken to, etc.


Vr interaction techniques4

VR interaction techniques

GEOPROVE: Geometric Probes for VEs

Making Measurements in VR

  • Uses probes consisting of markers


Vr interaction techniques5

VR interaction techniques

GEOPROVE: Geometric Probes for VEs


Let s look at an example

Let’s look at an example...


Simulated vascular reconstruction in a virtual environment

Simulated vascular reconstructionin a virtual environment


Overview1

Overview

  • Interactive virtual environments for the exploration of

    • Multi-dimensional datasets

    • Multi-parameter spaces (computational steering)

  • Visualization and interaction in Virtual Reality (VR)

  • Applied to a test case:simulated vascular reconstruction in VR


Interactive exploration in virtual environments

VRE

  • The Virtual Radiology Explorer(VRE):

    • Static exploration of 3D medical datasets

    • Virtual Reality (VR) interface

      • CAVE at SARA, Amsterdam

      • Portable ImmersaDesk

    • Surface/volume rendering

      • Virtual endoscopy

    • PACS data and computinginterface

      • Data storage and processingon parallel system (IBM SP2)


Vascular disease

Vascular disease

  • Stenosis:Treatment: thrombolysis, balloon angioplasty, stent placement, endarterectomy, bypass

  • Aneurysm:Treatment: shunt, bypass


The problem

The problem

  • Best treatment often not obvious

    • read: the parameter space

  • Human body is a complex structure

    • read: the data space

  • A treatment is not always best under all situations

    • read: combination of both


Pre operative planning

Pre-operative planning


Traditional treatment of vascular disease

Traditional treatment of vascular disease


Interactive simulated vascular surgery

Interactive simulated vascular surgery


Simulated vascular reconstruction

Simulated Vascular Reconstruction

  • Simulated vascular reconstruction

    • Patient specific angiographydata

    • Fluid flow simulationsoftware

    • Simulation of reconstructivesurgical procedure in VR

    • Interactive visualization ofsimulation results in VR

  • Pre-operative planning

    • Explore multiple reconstructionprocedures


Preprocessing

Preprocessing

  • Segmentation of patient specific MRA/CTA scan

    • Isolates region of interest

  • Lattice Boltzmann grid generation

    • Defines solid and fluid nodes, inlet and outlet conditions


Parallel fluid flow simulation

Parallel fluid flow simulation

  • Lattice Boltzmann Method (LBM)

    • Lattice based particle method

      • Regular lattice, similar to CT or MRI datasets

      • Allows irregular 3D geometry

      • Allows changes at run-time

  • Velocity, pressure and shearstress calculated fromparticle densities

  • Non-compressiblehomogeneous fluid,laminar flow

  • Spatial and temporallocality

    • Ideal for parallelimplementation


Parallel fluid flow simulation1

Parallel fluid flow simulation


Interactive exploration in vr

Interactive exploration in VR

  • Visualize simulation results

    • Flow field, pressure, shear stress

    • Real time


Interactive exploration in vr1

Interactive exploration in VR

  • Visualize simulation results

    • Flow field, pressure, shear stress

    • Real time


Interactive exploration in vr2

Interactive exploration in VR

  • Visualize simulation results

    • Flow field, pressure, shear stress

    • Real time

  • Interactive grid editing

    • Simulate vascularreconstruction procedure


Interactive exploration in vr3

Interactive exploration in VR

  • Visualize simulation results

    • Flow field, pressure, shear stress

    • Real time

  • Interactive grid editing

    • Simulate vascularreconstruction procedure

  • Interactive exploration

    • VR interaction to locateregions of interest


Interactive exploration in vr4

Interactive exploration in VR

  • Visualize simulation results

    • Flow field, pressure, shear stress

    • Real time

  • Interactive grid editing

    • Simulate vascularreconstruction procedure

  • Interactive exploration

    • VR interaction to locateregions of interest

  • Taking Measurements inVR


The future

The Future


The grid vs the www

The Grid .vs. the WWW

The WWW is about sharing information.

The Grid is about sharing resources.

databases, web pages, high performance computing systems, networks, data acquisition equipment, data storage / archiving and retrieval facilities, algorithms, search engines, images, …


The grid vs the www1

Applications

Applications

Applications

App Toolkit

App Toolkit

App Toolkit

The Grid .vs. the WWW

The WWW is about sharing information.

The Grid is about sharing resources.

Applications

Grid services layer

Grid “middleware”

Grid fabric layer


Virtual laboratory for e science

Applicationspecific

services

Application

genericservices

&

VL-eservices

Virtual Laboratory

rapid prototyping

(interactive simulation)

Additional

Grid Services

(OGSA services)

Grid

&

Network

Services

VL-eCertification

Network Services

(lambda networking)

VL-eExperimental Environment

Virtual Laboratory for e-Science

Bio-informatics

Medical

Applications

Telescience

Virtual Laboratory middleware

Grid Middleware

SURFnet

VL-eProof of ConceptEnvironment


Partners

Partners

Philips Intera 3T MRI scannerAMC, Amsterdam

MEG scannerVUmc, Amsterdam


Why grids why vl e

Why Grids? Why VL-e?

  • Integrated access to distributed computational resources, irrespective of time and/or location.For example:

    • High Performance Computing

      • High throughput, low-latency computing

    • High Capacity Storage

      • Remote PACS, shared data access

    • Medical image data and patient records

      • Longitudinal/epidemiological studies, atlases

    • Algorithms

      • Simulations, image processing, visualization

    • People

      • Collaboration, 2nd opinions, remote diagnostics

    • ...


Medical problem solving environment

2D/3D visualization

VL experiment topology

Image processing,Data storage

Medical Problem Solving Environment

Data retrieval,acquisition

Filtering, analyses,simulation


Msc projects

MSc projects

  • Interactive Distributed Computing Middleware

    • Linda (Tuple Spaces), HLA

  • Visualization subjects:

    • Applied interactive visualization:

      • Astrophysics, (bio)medical

    • Visualization grid/web services

    • Standards (X3D, Java3D)

    • GPUs …

      • … for high performance rendering

      • … for generic computing

  • Interaction subjects:

    • Interaction techniques

    • Device abstraction in user interaction

    • Usability studies

      For more info:

      http://www.science.uva.nl/~robbel/

      or email

      [email protected]


  • Login