Mixed reality systems
Sponsored Links
This presentation is the property of its rightful owner.
1 / 45

Mixed Reality Systems PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on
  • Presentation posted in: General

Mixed Reality Systems. Lab IV – Augmented Reality- Christoph Anthes. Overview. ARToolKit Combining ARToolKit with OpenSG Initialisation ARToolKit Loop Helper Functions Tracking Objects Interaction with objects Combining ARToolKit with inVRs. ARToolKit.

Download Presentation

Mixed Reality Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Mixed Reality Systems

Lab IV – Augmented Reality-

Christoph Anthes


Overview

  • ARToolKit

  • Combining ARToolKit with OpenSG

    • Initialisation

    • ARToolKit Loop

    • Helper Functions

    • Tracking Objects

    • Interaction with objects

  • Combining ARToolKit with inVRs

Mixed Reality Systems


ARToolKit

  • Current Version 2.72.1available at Sourceforge page

  • C and C++ API

  • Cross-platform API (Windows, Linux, MacOS, IRIX)

  • OpenGL used for rendering, GLUT used for event handling

  • Used video library depends on chosen platform

  • Architecture – General Picture

    • Build on GLUT with OpenGL, C and C++

    • Makes use of the video device specific and

      graphics drivers

    • Often used only as an independent tracking

      library

    • Own application is designed to be build on top of OpenGL and ARToolKit

    • Interfaces to larger tracking libraries exist (e.g. OpenTracker)

From http://www.hitl.washington.edu/artoolkit/documentation/

Mixed Reality Systems


ARToolKit

  • Architecture – ARToolKit Focus

    • AR module

      • Core module with marker

        tracking, calibration and

        parameter collection

    • Video module

      • Collection of video routines for

        capturing the video input frames

      • Wrapper around the standard

        platform SDK video capture routines

    • Gsub module

      • Graphic routines based on the OpenGL and GLUT libraries

    • Gsub_lite module

      • Replaces GSub with a more efficient collection of graphics routines

      • Independent of any particular windowing toolkit

From http://www.hitl.washington.edu/artoolkit/documentation/


ARToolKit

  • Coordinate Systems

    • Several coordinate

      systems are to be used

    • Most important camera and

      marker coordinates

    • From camera to screen coordinates

      a transformation via a distortion

      function can be performed

    • Z-axis of the marker is pointing upward

    • Z-axis of the camera is pointing in the scene

    • Top left corner of the screen is 0,0

  • arGetTransMat()

    • Returns the coordinates of the marker in the Camera coordinate System

  • arMatrixInverse()

    • Returns the coordinates of the camera in the marker coordinate system

From http://www.hitl.washington.edu/artoolkit/documentation/


ARToolKit

  • Basic Application

From http://www.hitl.washington.edu/artoolkit/documentation/

Mixed Reality Systems


ARToolKit

  • Corresponding Function Calls in plain ARToolKit API

From http://www.hitl.washington.edu/artoolkit/documentation/

Mixed Reality Systems


ARToolKit

  • Connections to Scene Graphs in General

    • Three steps

      • Initialising the Camera

      • Transforming the Object

      • “Real Occluders” – slightly advanced

    • OpenSG

      • Creation of a new node with fscEdit

    • OpenSceneGraph

      • Extension of scene view class

      • Display of occluding geometry

      • Overwrite colour buffer with video image

      • Finally display of recognised objects

  • Two big approaches

    • OSGART (http://www.artoolworks.com/community/osgart/index.html)

    • OSGAR (http://www.gvu.gatech.edu/ael/projects/ARSceneGraph.html)

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • OpenSG provides examples for interconnecting either ARToolKit or ARToolKit Plus

  • We are going to work with ARToolKit

  • So we first need the additional include files

  • gsub.h – contains main display functions used in ARToolkit

  • video.h – provides multi-platform video input support for ARToolKit

  • param.h – contains principal routines for loading, saving, and modify camera parameters

  • ar.h – provides image analysis and marker detection routines

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • If we take a look at our example we start with a set of forward declarations some which we have not seen in the previous OpenSG or inVRs tutorial

  • Additionally more methods are used at the end of the code

  • We start with the setup and cleanup methods:

    • initARToolkit() is used for initialisation of the ARToolKit components of the example

    • initOpenSG() initialises the OpenSG setup of the example

    • initGlut() registers the GLUT callbacks as we have seen in previous examples

    • setupCamera () provides a setup for our interconnected webcam

    • cleanupARToolkit() stops the video capture of ARToolKit and closes the video stream processing

    • cleanupOpenSG() frees used variables, stops the binding to ARToolKit, and call osgExit()

  • Let’s have a detailed look at the setup methods

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • initARToolkit()

    • Wraps and calls the different internal setup functions for ARToolKit

    • The cleanup method is registered as a callback at program termination

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • setupCamera()

    • The camera setup is defined in this function

    • Camera parameters from calibration file are parsed and evaluated

    • Conversion has to take place in order to write data out in the right format

    • The ModelViewMatrix and the ProjectionMatrix are set

    • This setup is stored inside an OpenSG camera object

  • The camera parameters are stored in binary files

    • E.g. currentParams.mat

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • setupCamera()

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • initOpenSG()

    • The standard OpenSG setup is performed

    • A GLUT window is created and initialised

    • A root node with an anonymous group core is created

    • The previously described camera setup is triggered

    • A background object is interconnected with a video texture

    • The SimpleSceneManager is initialised and interconnected with the window and the root node of the scene

    • The background image as well as the camera are attached to the viewport of the just generated window

    • All changes are committed to OpenSG

    • Finally the OpenSG cleanup function is registered to be triggered at the termination of the application

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • initOpenSG()

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • Then we have the ARToolKit processing methods

    • In the captureFrame() method the image is retrieved from the camera

    • The detectMarkers() function triggers marker detection and outputs the amount of found markers

    • applyMarkerTrans() applies the transformation from a given marker to an OpenSG transformation core

  • ARToolKit Loop

    • These steps represent our steps 2-4 from our ARToolKit loop

    • Step 1 was the initialisation given with the previous set of functions

    • Step 5 the rendering is performed by OpenSG

  • They have to be processed frame by frame, thus they are called inside the display loop of the application

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • captureFrame()

    • This method retrieves an image which was captured by ARToolKit from the incoming video stream

    • The image data is set in an OpenSG image and an update notification is issued

    • This frame will later on be rendered as a background image

    • It is as well used by ARToolKit for image processing which is performed in the next step

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • detectMarkers()

    • In this method the markers visible on the image are detected

    • The amount of detected markers as well as the markers themselves are returned by reference

    • A threshold parameter determines the binarisation of the image

    • Output describing the amount of detected markers as well as the ids of the markers is generated on the console

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • applyMarkerTrans()

    • Extracts the transformation information from a marker and applies it to a given OpenSG model

    • By using an STL map a binding between

    • The marker transformation is requested from ARToolKit via arGetMarkerTrans

    • It is then converted into an OpenSG matrix

    • If a marker is found in the map it becomes activated again and the transformation matrix just retrieved is applied on an OpenSG object

    • The changes are finally committed

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • applyMarkerTrans()

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • Additional methods which are used as helper functions and appear in the code

    • The method arMatrixToOSGMatrix() performs a data conversion from ARToolKit to OpenSG

    • The argConvGLcpara() method is used for conversion of ARToolKit camera parameters to OpenGL parameters

    • getTranslation() returns the translation vector of a transformation matrix

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • arMatrixToOSGMatrix()

    • Some very basic reformatting of an ARToolKit Matrix to an OpenSG Matrix is performed in this method

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • argConvGLcpara() is used to transform ARToolKit intrinsic camera parameters matrix format to an OpenGL matrix format

    • More details on camera calibration and the parameters are given in the computer vision class in Winter semester

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • Additional methods which are used as helper functions and appear in the code

    • createPattern() connects a marker with an OpenSG sub scene graph

    • createBackground() creates an image background based on the image gathererd from the ARToolKit Video stream

    • createModel() loads a sub scene graph from disk and equips it with an additional transformation code

  • Before we start coding let’s have a more detailed look at these methods

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • createPattern()

    • This helper method interconnects a pattern from ARToolKit with a sub scene graph provided by OpenSG, it uses createModel() as helper

    • The pattern is then registered at an STL map

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • createBackground()

    • This method creates the image background of OpenSG based on the data gathered from the ARToolKit video stream

Mixed Reality Systems


Combining ARToolKit and OpenSG

  • createModel()

    • This is a helper method which simply loads a model and attaches it to a node with a ComponentTransform core

  • Now we should know all necessary helper functions and will go on with the main function and the display function

Mixed Reality Systems


Tracking Objects

  • Our main function in this example is very simple since most of the processing is performed in the display loop

    • Initialisation of OpenSG and ARToolKit is performed

    • The video loop and the display loop are triggered

  • We now insert our first snippet into main in order to display an object on a marker

  • Compile and execute now

Snippet 1-1

Mixed Reality Systems


Tracking Objects

  • If we now take a closer look at our display loop we can see at the beginning our 3 ARToolKit steps

  • We should now enhance the scene by adding two objects and removing the first one

  • If you compile and execute you code now you should be able to see two objects

  • Now we want to perform some interaction between the objects

Snippet 2-1

Mixed Reality Systems


Tracking Objects

  • With the next snippet we check the proximity of the two objects and change the scale

  • First we retrieve the transformations of the objects and calculate the distance

Snippet 2-2 – First Part

Mixed Reality Systems


Interaction with Objects

  • We calculate the scale based on the distance and apply it on the transformation matrices of our two objects

  • Afterwards changes have to be committed

Snippet 2-2 – Second Part

Mixed Reality Systems


Interaction with Objects

  • If you execute your code now you should see something like this (most likely with a different user)

Mixed Reality Systems


Interaction with Objects

  • In the next step things will become more complicated an object should move from a marker to another marker

  • Actually we want to have a frog jump from one stone on a marker to another stone on a different marker

  • First we uncomment the scaling part of the object by inserting the following snippet

Snippet 3-1

Mixed Reality Systems


Interaction with Objects

  • We initialise a frog model and other nodes for our animation

Snippet 3-2

Mixed Reality Systems


Interaction with Objects

  • Now we trigger the initialisation in the main method

  • For the animation of motion between one and another stone we install a timer

  • Inside the Display loop delta values are calculated in order to determine the time since the last frame

  • An overall counter is incremented

Snippet 3-3

Snippet 3-4

Mixed Reality Systems


Interaction with Objects

  • For animating a jumping frog from one stone to another stone, we implement a state machine with 5 states

    • SITTING – the frog sits for a given time before it can try to jump, afterwards it automatically switches to the WAITING state

    • WAITING – two markers have to be close enough to make the frog jump, if the proximity check returns positively it will jump

    • JUMPING – in the state a change in the scene graph is performed. The frog node is attached to a transformation node

    • FLYING – in this state the actual calculation for the trajectory of the frog is calculated and applied on the transformation node. Once the target is reached the state changes to LANDING

    • LANDING – as in the JUMPING state a change in the scene graph hierarchy is performed. The state machine switches to SITTING again.

  • The whole state machine is implemented inside a single snippet through which we will go through now

Mixed Reality Systems


Interaction with Objects

  • State machine – SITTING – WAITING

    • We check the passed time in sitting state and switch after 2000ms to the WAITING state. The counter is reset afterwards.

    • In the WAITING state we check the marker proximity and switch to the JUMPING state if successful.

Snippet 3-5 Part 1

Mixed Reality Systems


Interaction with Objects

  • State machine – JUMPING

    • We detach the frog from the stone and attach it to a transformation core the transformation matrix of the core is initialised with the markers transformation matrix

    • The state changes to FLYING and the counter is reset.

Snippet 3-5 Part 2

Mixed Reality Systems


Interaction with Objects

  • State machine – FLYING

    • Retrieve the target position and the current position and determine the path of the frog

    • In case the flying time becomes to high we switch back to the LANDING state

    • If the distance between the frog and the target becomes to big it is going back

Snippet 3-5 Part 3

Mixed Reality Systems


Interaction with Objects

  • State machine – FLYING continued

    • If everything went alright we determine the direction vector by normalising the path

    • We scale this normalised vector to move the frog at a constant speed based on the current deltaTime

    • No we calculate a transformation matrix based on this vector

    • Finally the generated transformation is applied on the frog transformation

Snippet 3-5 Part 4

Mixed Reality Systems


Interaction with Objects

  • State machine – LANDING

    • The frog is detached from the transformation node and attached to the new marker

    • The counter is reset and the state is set back to the initial SITTING state

    • At the end of each iteration of the display loop the changes are committed

Snippet 3-4 Part 5

Mixed Reality Systems


Interaction with Objects

  • This is how it should look

  • The final step is fully up to

    you to implement

  • Have a frog jumping

    between three markers

    • You will have to load an

      additional marker

    • You will have to determine

      the distance between the

      three different markers and

      determine the closest

    • Let the frog hop from one marker to the closest marker

Mixed Reality Systems


Combining ARToolKit with inVRs

  • inVRs provides a simple binding to the ARToolKit which implements interaction with markers and the scene

  • The next lab will show some basic desktop interaction with the ARToolKit

  • To enable full AR support for the inVRs framework the input interface as well as the output interface can be enhanced

  • Things to do at home

    • Try to make two frogs switch stones when they look at each other

    • Take a look at multi-pattern calibration

    • Try to interconnect inVRs with the ARToolKit and write an ARToolKit marker input device

    • Take a look at ARToolKit Plus, what are the advantages?

    • Can you manage to interconnect ARToolKit Plus with OpenSG or inVRs?

Mixed Reality Systems


Useful Links

ARToolKit

http://artoolkit.sourceforge.net/ - Sourceforge Entry page

http://www.hitl.washington.edu/artoolkit/ - GPL version

http://www.artoolworks.com/ - commercial version

ARToolKit Plus Web Page

http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php

OpenSG Web Page

http://www.opensg.org/

inVRs Web Page

http://www.invrs.org/

Mixed Reality Systems

44


Thank You !


  • Login