Underlying technologies part two software
This presentation is the property of its rightful owner.
Sponsored Links
1 / 65

Underlying Technologies Part Two: Software PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Underlying Technologies Part Two: Software. Mark Green School of Creative Media. Introduction. Software not as easy as hardware: wide range of software techniques, hard to classify like hardware several components that need to work together, hard to know where to start

Download Presentation

Underlying Technologies Part Two: Software

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Underlying technologies part two software

Underlying TechnologiesPart Two: Software

Mark Green

School of Creative Media



  • Software not as easy as hardware:

    • wide range of software techniques, hard to classify like hardware

    • several components that need to work together, hard to know where to start

    • wide range of hardware configurations, not as simple as 2D software

Hardware configurations

Hardware Configurations

  • In 2D have a standard hardware configuration:

    • input: keyboard and mouse

    • output: single 2D display

  • with 3D can have many configurations:

    • HMD

    • projection

    • single screen

Hardware configurations1

Hardware Configurations

  • Want to produce an application once, not once for every possible hardware configuration

  • software needs to be more adaptable, change based on hardware configuration

  • complicates the development of support software

Range of software techniques

Range of Software Techniques

  • Want our software to be very efficient: reduce latency, high update rates

  • some applications can be quite large, need to efficiently organize data

  • all of this complicates VR software, too many things to consider, hard to know where to start



  • What are the main components of a VR application?

    • 3D Objects: geometry and appearance, but may also want sound and force

    • Behavior: the objects need to be able to do things, move and react

    • Interaction: users want to interact with the application, manipulate the objects

3d objects

3D Objects

  • Need object geometry, object’s shape, basis for everything else, called model

  • polygons used for geometry, sometimes restricted to triangles

  • different from animation, free form surfaces based on sophisticated math

  • need speed, so restricted to polygons

3d objects1

3D Objects

  • Where does geometry come from?

  • Really depends on the application

  • Could use a text editor to enter all the polygon vertices, some people actually do this!

  • Could use a program, for example OpenGL, works for small models

3d objects2

3D Objects

  • Use a 3D modeling or animation program

  • for non-programmers this is the easiest way, but it takes time to develop modeling skills

  • also many different program and file formats

  • want a modeler that does a good job of polygons, not all modelers are good at this

3d objects3

3D Objects

  • Another source of objects is scientific and engineering computations

  • can be easy to convert to polygons, already have position data

  • other types of data can also be converted into geometry, but this can be more difficult

3d objects4

3D Objects

  • Also need to consider appearance:

    • colour of the object

    • how it reflects light

    • transparency

    • texture

  • can be done with modeler, or later in the VR program



  • How should objects behave?

    • What happens when the user hits an object?

    • What happens when an object hits another object?

    • Can objects move around the environment?

  • Each object could have a range of behaviors, react differently to different events in the environment



  • Behavior is harder than modeling

  • animation programs can be useful, but not always

  • animation is quite different:

    • animator is in complete control, knows what’s happening all of the time

    • in VR the user is in control, can interrupt or mess up any animation



  • Short animations (less than 5 seconds) can be useful, basic motion units

  • other types of behaviors must be programmed or scripted

  • more flexible, can respond to the events that occur in the environment

  • easier to combine, objects can do two things at same time



  • Users want to interact with the environment

  • pick up objects and move them around

  • very different from 2D interaction

  • much more freedom, more direct interaction

  • still exploring the design space, not stable like 2D interaction

  • still working on standard techniques

Application structure

Application Structure

  • look at application structure

  • provides a framework for discussing various software technologies

  • divide an application into various components, and then look at the components individually

Application structure1

Application Structure








Model Traversal

Output Devices

Application structure2

Application Structure

  • Model: representation of objects in the environment, geometry and behavior

  • Traversal: convert the model into graphical, sound, force, etc output

  • Input Processing: determine user’s intentions, act on other part of application

  • application processing: non-VR parts of the application

Interaction loop

Interaction Loop

  • Logically the program consists of a loop that samples the user, performs computations and traverses the model

Input processing


Model Traversal



  • Contains the information required to display the environment:

    • geometry, sound, force

    • behavior

  • the graphical part is the most developed, so concentrate on it

  • try to position sound and force within this model



  • This is what we know the best

  • need to have a graphical representation of objects in the environment:

    • accurate shape representation

    • ease of modeling

    • efficient display

    • integrates with behavior

Scene graph

Scene Graph

  • Main technique for structuring the model

  • based on hierarchical structure, divide the object into parts or components

  • simplifies the modeling task, work on one part at a time

  • easy to modify the individual parts

  • add behaviors, sound, force, etc to the model

Scene graph1

Scene Graph







Scene graph2

Scene Graph

  • Individual units are called nodes:

    • shapes: polygons, meshes, cubes, etc

    • transformations: position the nodes in space

    • material: colour and texture of objects

    • grouping: collecting nodes together as a single object

    • sounds

    • behavior

Scene graph3

Scene Graph

  • Many different scene graph architectures, will look at one in more detail later

  • differences:

    • scene graph for whole VE Vs. one per object

    • types of nodes in the scene graph

    • ease of modification, static Vs dynamic



  • Harder to deal with than geometry

  • simple motions aren’t too bad, but much harder to get sophisticated behavior

  • the general solution now is to write code, okay for programmers

  • would like to have a higher level approach for non-programmers



  • Problem: want objects to respond to events in the environment

  • can have some motions that are simple animations, but most of the motions need some knowledge of the environment

  • example: an object moving through the environment must be aware of other objects so it doesn’t walk through them



  • Some simple motions produced by animating transformation nodes

  • animation variables used to control transformation parameters, example: rotation or translation

  • could import animations, use some form of keyframing package to produce the motion



  • Simple motions could be triggered by events in the environment

  • example: collision detection, if an object is moving through the environment and a collision detected it changes direction

  • hard to come up with good trigger conditions, a few obvious ones, but not much after that



  • Another approach is to use a general motion model

  • best example of this is physics, try to simulate real physics in the environment

  • this gives a number of natural motions, and objects respond to the environment

  • works well in some environment, but has some problems



  • One problem is the complexity of the mathematics, often need to simplify

  • computations can be a problem, particularly for complex objects

  • hard to control, need to know forces and torque's that produce the desired motions, can be very hard to determine



  • Some attempts to produce general motion controllers

  • maybe the eventual solution, but nothing much now

  • use of scripting languages, can add some program control to the scene graph, but not full programming

Model traversal

Model Traversal

  • The process of going through the model and generating the information to be displayed

  • this is part software and part hardware, look through the entire process

  • hardware parts have implications for how we build models and the graphics techniques used

A simple model

A Simple Model

  • A simplified model of the display process, explains hardware performance








  • Traverse the model, determine objects to be drawn, send to graphics hardware

  • usually combination software/hardware, depends on CPU and bus speed

  • early systems were hardware, didn’t scale well

  • many software techniques for culling models



  • Geometrical computations on polygons: transformations and lighting

  • floating point intensive

  • divide polygons into fragments, screen aligned trapezoid

  • time proportional to number of polygons and vertices



  • Fill fragments, colour interpolation, texture mapping, transparency, hidden surface

  • all the per pixel computations, time depends on number of pixels, also colour depth on low end displays

  • scalable operations, can add more processors for more speed

Design considerations

Design Considerations

  • Any of the stages could block, depend on display mix

  • lots of small polygons cause traversal and geometry stages to block

  • large polygons cause pixel stage to block

  • can use buffers to reduce blocking

  • a careful balancing process

Design considerations1

Design Considerations

  • CPU/Image Generator trade-off

  • cheap boards just do pixel stage, use CPU for everything else:

    • scales with CPU speed

    • large polygons and texture mapping

  • moving geometry onto board increases performance, trend in low cost displays

Pc hardware evolution

PC Hardware Evolution

  • Start with CPU doing most of the work

  • Graphics board:

    • image memory

    • fill and hidden surface

    • texture mapping

  • graphics speed determined by CPU, limited assistance from graphics card

Graphics card memory

Graphics Card Memory

  • Memory used for three things:

    • image store

    • hidden surface (z buffer)

    • texture maps

  • texture can be stored in main memory with AGP, but this isn’t most efficient

  • better to have texture memory on board

Image memory

Image Memory

  • Amount depends on image size

  • double buffer, two copies of image memory

    • front buffer: image displayed on screen

    • back buffer: where the next image is constructed

  • can construct next image while the current image is displayed, better image quality and faster display

Z buffer

Z Buffer

  • Used for hidden surface removal

  • z buffer: one value for each pixel, distance from eye to object drawn at that pixel

  • when drawing a pixel, compare depth of pixel to z buffer

  • if closer draw pixel and update z buffer

  • otherwise, ignore the pixel

Graphics acceleration

Graphics Acceleration

  • Next step: move pixel operations to graphics card

  • fill and z buffer 3D triangles

  • add smooth shading and texture mapping

  • CPU does traversal and geometry processing

Graphics acceleration1

Graphics Acceleration

  • Next step: move geometry processing to graphics card

  • CPU traverses model, send graphics primitives to display card

  • all transformations and lighting done on graphics card

  • less dependence on CPU

Current trends

Current Trends

  • Pixel processing (Geforce 2): a program that processes each pixel, control lighting and other effects

  • support for multiple textures, etc

  • Vertex processing (Geforce 3): a program processes each vertex, can change geometry at display time

  • real-time deformations and IKA

Current trends1

Current Trends

  • Move to programming all aspects of the graphics card (3DLabs VP series)

  • Also making programming more sophisticated, closer to CPU

  • Floating point textures and image memory (ATI and 3DLabs VP series)

  • Higher dynamic range -> better image quality, better for programming

Input processing

Input Processing

  • Users need to interact with the environment

  • they have a set of input devices, have position and orientation information

  • need to translate this into their intentions

  • Interaction Technique (IT): basic unit of interaction, converts user input into something the application understands

Input processing1

Input Processing

  • Each IT address a particular interaction task, something that the user wants to do

  • look at interaction tasks first, then talk a little bit about ITs for them

  • interaction tasks divide into two groups:

    • application independent: required by many different applications

    • application dependent

Interaction tasks

Interaction Tasks

  • Mainly look at application independent interaction tasks

  • the main ones are:

    • navigation

    • selection

    • manipulation

    • combination



  • Need to get from one part of the environment to another

  • two types:

    • local

    • global

  • with local navigation the destination is within view, move on continuous path from current location to destination



  • In global navigation the destination is remote, can’t move directly to it

  • need some way of locating destination, and then some way of jumping to it

  • variation: browsing / exploration don’t have a destination, exploring the environment or looking for particular objects



  • The selection tasks involves selecting something

  • there are several variations, depending upon what’s being selected:

    • list or command selection

    • object selection

    • location selection



  • List selection: a pre-defined list of things to select from

  • example: the commands on a menu

  • need to present the list, and the user selects one item from the list

  • object selection: number of objects not pre-defined, created by the user, changes in size as the program runs



  • For object selection can’t just present a list of objects to be selected from

  • location selection: selecting a point in space, may be used as location of object, or as part of an object’s shape

  • can’t see a point in empty space, so this is harder than the previous two



  • Standard set of object manipulations, change position, size and orientation

  • grab the object and move it

  • could also have deformations that change the object’s shape

  • hard to get general techniques beyond the few standard ones



  • Take two or more objects and put them together to form a new object

  • need to match up the shapes exactly, so they join in the right way

  • difficult to do unaided, usually need some form of constraint to simplify the process

Application dependent tasks

Application Dependent Tasks

  • Usually involve the application data

  • ways of controlling the view of the data

  • ways of manipulating the data

  • example: a CAD or animation program will have a different set of manipulations than a network visualization program

Interaction techniques

Interaction Techniques

  • Not a well established set of techniques, yet

  • depend on input devices and style

  • example: a fixed range device (tracker) sometimes works best with pointing at objects, while a puck or joystick might work better with grabbing

  • need to try different combinations

Interaction techniques1

Interaction Techniques

  • Some problems encountered:

    • distance: objects too far away to grab

    • feedback: how do you highlight the object that has been selected?

    • Object to be selected may be hidden by other objects

    • object density may make selection and manipulation difficult

Application processing

Application Processing

  • Not much to say here

  • some applications have a considerable amount of processing, computation based on user input

  • don’t want this to effect application latency

  • need to control resources devoted to computation, use other processors

Making it run right

Making it run right

  • Now that we have an idea of what’s involved, how do we put it all together

  • want to have low system latency, get images on the screen as fast as possible

  • don’t want to wait for anything

  • divide the application into components that execute separately

Decoupled simulation model

Decoupled Simulation Model

  • Separate process for application computations, this is easy

  • separate process for expensive input devices, trackers that need lots of computation or have latency

  • a separate process for input processing and display

  • maybe a separate process for model

Application structure3

Application Structure








Model Traversal

Output Devices

Decoupled simulation model1

Decoupled Simulation Model

  • Each process can run at its own rate

  • display process runs as fast as possible, doesn’t wait for other processes

  • uses most recent value from input devices and application computation

  • reduces system latency

  • Login