slide1
Download
Skip this Video
Download Presentation
A Framework for Multi-resolution and Multi-touch Systems

Loading in 2 Seconds...

play fullscreen
1 / 42

A Framework for Multi-resolution and Multi-touch Systems - PowerPoint PPT Presentation


  • 91 Views
  • Uploaded on

A Framework for Multi-resolution and Multi-touch Systems . 多重解析度及多重觸控互動系統之開發. Presenter: 胡婷婷. Advisor: 洪一平 教授. Outline. Introduction Related Work i -m- Top System i -m- Top SDK Application Conclusion & Future Work. Outline. Introduction Related Work i -m- Top System

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' A Framework for Multi-resolution and Multi-touch Systems ' - hisa


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

A Framework for

Multi-resolution and Multi-touch Systems

多重解析度及多重觸控互動系統之開發

Presenter: 胡婷婷

Advisor: 洪一平 教授

outline
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
outline1
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
introduction
Introduction
  • Motivation
    • Multi-resolution
      • Variable-resolution nature of human vision
      • Cost-effective large high-res display
    • Multi-touch
      • Intuitive interaction
      • Computer-vision based approach
    • Software Toolkit
      • Hide engineering challenges
introduction1
Introduction
  • i-m-Top
    • interactive multi-resolution tableTop
      • interactive: multi-touch
      • multi-resolution: fovea + peripheral projectors
  • i-m-Top SDK
    • Toolkit for rapid prototyping applications
      • Featured on multi-resolution/multi-touch
introduction2
Introduction
  • Overview

Cams

Image Processing

Applications

i-m-Top

SDK

Projectors

Touch Detection

Pan-tilt Unit

Multimedia

Other sensors

User Interface

outline2
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
related work
Related Work
  • Multi-Resolution Display
    • Fovea Plus Context
      • LCD screen (fixed) + projector

Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST \'01

related work1
Related Work
  • Multi-Resolution Display
    • Fovea-Tablett
      • Tablets as high-res displays
        • Tracked with visual marker
        • Multiple high-res is allowed
      • Cons: physical boundary

J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea-tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007

related work2
Related Work
  • Software Development Toolkit
    • DiamondTouch SDK (C++)
      • Multi-touch, multi-user

Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002

related work3
Related Work
  • Software Development Toolkit
    • DiamondSpin(Java):
      • Around-the-table interaction
        • Real-time polar to Cartesian transformation

rotatable circular tabletop

public + personal work areas

Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004

related work4
Related Work
  • Software Development Toolkit
    • T3 (Java)
      • Multiple projectors → single high-res display
      • Mixed-presence collaboration

Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.

outline3
Outline
  • Introduction
  • Related Work
  • i-m-Top System
    • Hardware Configuration
    • Touch Detection
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
i m top system
i-m-Top System

96 ppi

24 ppi

56”

Mirror

Pan-tilt Unit

Fovea Projector

IR Camera

Peripheral Projector

IR Illuminator

i m top system2
i-m-Top System
  • Touch Detection

finger

palm

Opening

Background Subtraction

Connected

Component

PCA

Analysis

Palm-finger

Association

Finger

Fusion

Finger Tracking

-

-

Background Subtraction

Connected

Component

PCA

Analysis

Palm

Association

Opening

Opening

Down-

scale

Connected

Component

Palm

Tracking

-

outline4
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
    • Design Principles
    • Architecture
    • Implementation
  • Application
  • Conclusion & Future Work
i m top sdk
i-m-Top SDK
  • Design Principles
    • Efficient prototyping
      • Hide engineering details
      • Use metaphors
      • Provide reusable components
    • Touch-detection module independent
      • Vision-based & electronic-based approaches
    • Real-time requirement
      • Touch event handling + UI rendering
    • Extensibility
      • Integrate with other components
i m top sdk1
i-m-Top SDK
  • Architecture

Virtual Director

Scene

Scene

Scene

Scene

Node

Node

View

View

Node

Node

Viewport /

Virtual Camera

Global Coordinate

Position /

Textures /

Event Handlers /

Animation Effects

Node

Virtual Camera

(Fovea View)

Node

Virtual Camera

(Peripheral View)

Local Coordinate

i m top sdk2
i-m-Top SDK
  • Architecture

Application

Projector

Virtual Director

Touch

Detection

Projector

Message

Parser

Scene

Device

Control

Socket

Event

queue

Node

View

PTU

Window Message

Speaker

Node

Cam

Other

Sensors

COM

LED Light

OpenGL c#

i m top sdk3
i-m-Top SDK
  • Loop of virtual director

Time spent

0

Affiliated Timer Functions

Check Inputs

Update States

T

Render Scenes

Multimedia Output

1/fps-T

Sleep

1/fps

i m top sdk4
i-m-Top SDK
  • Sample Code
    • A scene with a photo
    • InitScenes( ){
    • oScene scene = newoScene(); //construct a scene
    • director.AddScene(scene); //add scene to director
    • oView view = newView(0, 0, 1280, 720); //construct a view
    • scene.AddView(view); //add view to scene
    • oPhoto photo = newoPhoto(0,0,-100,640,480,”sun.png”); //construct a photo
    • scene.AddNode(photo); //add to scene
  • }
i m top sdk5
i-m-Top SDK
  • Implementation
    • Multi-resolution Handling
    • Touch-event Processing
    • Reusable Components
implementation
Implementation
  • Multi-resolution Handling
    • Calibration
      • Homography of fovea & peripheral projections
      • Run-time warping
    • Masking
    • Steerable fovea projection
      • Region/Object oriented
    • Multi-presented Information
implementation1
Implementation
  • Calibration
    • Homography of fovea & peripheral projections
      • Provide tools for manual calibration
      • Save homography/ center point /pan-tilt of PTU

P2

P1

V1

V2

Y

Y

F1

F1

F2

F2

V1

V2

X

V1

V2

X

V3

V4

V3

V4

F3

F3

F4

F4

V3

V4

P3

P4

implementation2
Implementation
  • Calibration
    • Run-time warping
      • Adapt OpenGL rendering pipeline

Warping Homography Matrix

Model-View Matrix

Projection Matrix

Perspective Division

Viewport Transformation

Local coordinate

Global coordinate

Eye

coordinate

Clipping coordinate

Window

coordinate

Normalized device coordinate

implementation3
Implementation
  • Masking

fovea

peripheral

fovea

implementation4
Implementation
  • Masking
    • Find maximum region in fovea
      • Utilize Stencil buffer in OpenGL

Apply

arbitrary masks

Set ‘0’ in mask regions

Invert pixels in fovea region

Set ‘1’ in the fovea region

implementation5
Implementation
  • Steerable Fovea Projection
    • Region-oriented
      • Spotlight on a specific region
      • Steered by finger gestures
    • Object-oriented
      • Spotlight on a specific object
      • Move with object
    • Real-time update
      • Steer the mirror
      • Retrieve homography
      • Apply mask
implementation6
Implementation
  • Multi-presented Information
    • Different content in fovea and peripheral regions
      • Eg. Google Map
      • Automatically change

oRectanglemulti_presentation =newoRectangle(0, 0,-100, 200, 200);

multi_presentation.LoadTexture(“context.png");//load low-resolution image

multi_presentation.LoadHighTexture(“detail.png");//load high-resolution image

implementation7
Implementation
  • Touch-event Processing

Touch

events

Touch

Detection

i-m-Top SDK

Message

Parser

Virtual

Director

Node

Receive

Pass

Check

Dispatch

Handle

(by event handler)

implementation8
Implementation
  • Touch-event Processing
    • Receive & parse
      • Run on background threads
    • Pass to director
      • Gather all events in one detection frame before pass
        • if rendering rate < detection rate
          • Keep vital events: ‘Up’, ‘Down’
          • Discard other events: ‘Move’

i-m-Top SDK

Message

Parser

Virtual

Director

Node

Pass

Receive

implementation9
Implementation
  • Touch-event Processing
    • Check
      • ‘Down’: Test if hit some node
        • By SELECTION mode in OpenGL
        • If yes: add to TargetTable (event id → node)
      • ‘Move’, ‘Up’: Check if there is a target
        • Hit TargetTable by event id
        • ‘Up’: Remove from TargetTable
    • Dispatch
      • Dispatch events to each node

i-m-Top SDK

Message

Parser

Virtual

Director

Node

Dispatch

Check

implementation10
Implementation
  • Touch-event Processing
    • Define event handler
      • Node decides how to act
      • Default handler
        • Move/Scale/Rotate under ‘Move’ events
      • Redefine handler
    • InitScenes( ){
    • //……
    • //add new event handlers to the photo object
  • photo.MultiFingerEvent += newMultiFingerHandler(MyMultiFingerHandler);
  • photo.FingerClick += newFingerClickHandler(MyFingerClickHandler);
  • }

i-m-Top SDK

Message

Parser

Virtual

Director

Node

Handle

implementation11
Implementation

P3

P3

P1

P2

P4

P4

Translation: T

Scaling: S

P3

P3

P1

P34

Rotation: R

P12

P2

P4

P4

implementation12
Implementation
  • Reusable components
    • Multimedia
      • Video/audio
      • Camera capture
    • UI components
      • Virtual keyboard
      • Text-pad
      • Button, scrollbar…
outline5
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
application
Application
  • Map Browser
    • Different content in the fovea & peripheral
  • Photo Browser
    • Object/Region oriented fovea projection
  • Emotion Player
    • Interactive art

c

outline6
Outline
  • Introduction
  • Related Work
  • i-m-Top System
  • i-m-Top SDK
  • Application
  • Conclusion & Future Work
conclusion
Conclusion
  • i-m-Top
    • Multi-resolution
      • Accommodating to human vision
      • Cost-effective large high-res display
    • Multi-touch
      • Intuitive interaction
  • i-m-Top SDK
    • Hide engineering challenges
      • Rapid prototyping applications
      • Investigating new interfaces / interactions
future work
Future Work
  • Attentive computing
    • Combine with gaze-tracking
  • 3D interaction
    • Surface to space
  • Auto calibration
    • Vision-based
    • Continuous pan-tilt / homographymap
ad