A Framework for
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

A Framework for Multi-resolution and Multi-touch Systems PowerPoint PPT Presentation


  • 40 Views
  • Uploaded on
  • Presentation posted in: General

A Framework for Multi-resolution and Multi-touch Systems . 多重解析度及多重觸控互動系統之開發. Presenter: 胡婷婷. Advisor: 洪一平 教授. Outline. Introduction Related Work i -m- Top System i -m- Top SDK Application Conclusion & Future Work. Outline. Introduction Related Work i -m- Top System

Download Presentation

A Framework for Multi-resolution and Multi-touch Systems

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A framework for multi resolution and multi touch systems

A Framework for

Multi-resolution and Multi-touch Systems

多重解析度及多重觸控互動系統之開發

Presenter: 胡婷婷

Advisor: 洪一平 教授


Outline

Outline

  • Introduction

  • Related Work

  • i-m-Top System

  • i-m-Top SDK

  • Application

  • Conclusion & Future Work


Outline1

Outline

  • Introduction

  • Related Work

  • i-m-Top System

  • i-m-Top SDK

  • Application

  • Conclusion & Future Work


Introduction

Introduction

  • Motivation

    • Multi-resolution

      • Variable-resolution nature of human vision

      • Cost-effective large high-res display

    • Multi-touch

      • Intuitive interaction

      • Computer-vision based approach

    • Software Toolkit

      • Hide engineering challenges


Introduction1

Introduction

  • i-m-Top

    • interactive multi-resolution tableTop

      • interactive: multi-touch

      • multi-resolution: fovea + peripheral projectors

  • i-m-Top SDK

    • Toolkit for rapid prototyping applications

      • Featured on multi-resolution/multi-touch


Introduction2

Introduction

  • Overview

Cams

Image Processing

Applications

i-m-Top

SDK

Projectors

Touch Detection

Pan-tilt Unit

Multimedia

Other sensors

User Interface


Outline2

Outline

  • Introduction

  • Related Work

  • i-m-Top System

  • i-m-Top SDK

  • Application

  • Conclusion & Future Work


Related work

Related Work

  • Multi-Resolution Display

    • Fovea Plus Context

      • LCD screen (fixed) + projector

Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01


Related work1

Related Work

  • Multi-Resolution Display

    • Fovea-Tablett

      • Tablets as high-res displays

        • Tracked with visual marker

        • Multiple high-res is allowed

      • Cons: physical boundary

J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea-tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007


Related work2

Related Work

  • Software Development Toolkit

    • DiamondTouch SDK (C++)

      • Multi-touch, multi-user

Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002


Related work3

Related Work

  • Software Development Toolkit

    • DiamondSpin(Java):

      • Around-the-table interaction

        • Real-time polar to Cartesian transformation

rotatable circular tabletop

public + personal work areas

Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004


Related work4

Related Work

  • Software Development Toolkit

    • T3 (Java)

      • Multiple projectors → single high-res display

      • Mixed-presence collaboration

Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.


Outline3

Outline

  • Introduction

  • Related Work

  • i-m-Top System

    • Hardware Configuration

    • Touch Detection

  • i-m-Top SDK

  • Application

  • Conclusion & Future Work


I m top system

i-m-Top System

96 ppi

24 ppi

56”

Mirror

Pan-tilt Unit

Fovea Projector

IR Camera

Peripheral Projector

IR Illuminator


I m top system1

i-m-Top System


I m top system2

i-m-Top System

  • Touch Detection

finger

palm

Opening

Background Subtraction

Connected

Component

PCA

Analysis

Palm-finger

Association

Finger

Fusion

Finger Tracking

-

-

Background Subtraction

Connected

Component

PCA

Analysis

Palm

Association

Opening

Opening

Down-

scale

Connected

Component

Palm

Tracking

-


Outline4

Outline

  • Introduction

  • Related Work

  • i-m-Top System

  • i-m-Top SDK

    • Design Principles

    • Architecture

    • Implementation

  • Application

  • Conclusion & Future Work


I m top sdk

i-m-Top SDK

  • Design Principles

    • Efficient prototyping

      • Hide engineering details

      • Use metaphors

      • Provide reusable components

    • Touch-detection module independent

      • Vision-based & electronic-based approaches

    • Real-time requirement

      • Touch event handling + UI rendering

    • Extensibility

      • Integrate with other components


I m top sdk1

i-m-Top SDK

  • Architecture

Virtual Director

Scene

Scene

Scene

Scene

Node

Node

View

View

Node

Node

Viewport /

Virtual Camera

Global Coordinate

Position /

Textures /

Event Handlers /

Animation Effects

Node

Virtual Camera

(Fovea View)

Node

Virtual Camera

(Peripheral View)

Local Coordinate


I m top sdk2

i-m-Top SDK

  • Architecture

Application

Projector

Virtual Director

Touch

Detection

Projector

Message

Parser

Scene

Device

Control

Socket

Event

queue

Node

View

PTU

Window Message

Speaker

Node

Cam

Other

Sensors

COM

LED Light

OpenGL c#


I m top sdk3

i-m-Top SDK

  • Loop of virtual director

Time spent

0

Affiliated Timer Functions

Check Inputs

Update States

T

Render Scenes

Multimedia Output

1/fps-T

Sleep

1/fps


I m top sdk4

i-m-Top SDK

  • Sample Code

    • A scene with a photo

  • InitScenes( ){

  • oScene scene = newoScene();//construct a scene

  • director.AddScene(scene);//add scene to director

  • oView view = newView(0, 0, 1280, 720);//construct a view

  • scene.AddView(view);//add view to scene

  • oPhoto photo = newoPhoto(0,0,-100,640,480,”sun.png”);//construct a photo

  • scene.AddNode(photo);//add to scene

  • }


  • I m top sdk5

    i-m-Top SDK

    • Implementation

      • Multi-resolution Handling

      • Touch-event Processing

      • Reusable Components


    Implementation

    Implementation

    • Multi-resolution Handling

      • Calibration

        • Homography of fovea & peripheral projections

        • Run-time warping

      • Masking

      • Steerable fovea projection

        • Region/Object oriented

      • Multi-presented Information


    Implementation1

    Implementation

    • Calibration

      • Homography of fovea & peripheral projections

        • Provide tools for manual calibration

        • Save homography/ center point /pan-tilt of PTU

    P2

    P1

    V1

    V2

    Y

    Y

    F1

    F1

    F2

    F2

    V1

    V2

    X

    V1

    V2

    X

    V3

    V4

    V3

    V4

    F3

    F3

    F4

    F4

    V3

    V4

    P3

    P4


    Implementation2

    Implementation

    • Calibration

      • Run-time warping

        • Adapt OpenGL rendering pipeline

    Warping Homography Matrix

    Model-View Matrix

    Projection Matrix

    Perspective Division

    Viewport Transformation

    Local coordinate

    Global coordinate

    Eye

    coordinate

    Clipping coordinate

    Window

    coordinate

    Normalized device coordinate


    Implementation3

    Implementation

    • Masking

    fovea

    peripheral

    fovea


    Implementation4

    Implementation

    • Masking

      • Find maximum region in fovea

        • Utilize Stencil buffer in OpenGL

    Apply

    arbitrary masks

    Set ‘0’ in mask regions

    Invert pixels in fovea region

    Set ‘1’ in the fovea region


    Implementation5

    Implementation

    • Steerable Fovea Projection

      • Region-oriented

        • Spotlight on a specific region

        • Steered by finger gestures

      • Object-oriented

        • Spotlight on a specific object

        • Move with object

      • Real-time update

        • Steer the mirror

        • Retrieve homography

        • Apply mask


    Implementation6

    Implementation

    • Multi-presented Information

      • Different content in fovea and peripheral regions

        • Eg. Google Map

        • Automatically change

    oRectanglemulti_presentation =newoRectangle(0, 0,-100, 200, 200);

    multi_presentation.LoadTexture(“context.png");//load low-resolution image

    multi_presentation.LoadHighTexture(“detail.png");//load high-resolution image


    Implementation7

    Implementation

    • Touch-event Processing

    Touch

    events

    Touch

    Detection

    i-m-Top SDK

    Message

    Parser

    Virtual

    Director

    Node

    Receive

    Pass

    Check

    Dispatch

    Handle

    (by event handler)


    Implementation8

    Implementation

    • Touch-event Processing

      • Receive & parse

        • Run on background threads

      • Pass to director

        • Gather all events in one detection frame before pass

          • if rendering rate < detection rate

            • Keep vital events: ‘Up’, ‘Down’

            • Discard other events: ‘Move’

    i-m-Top SDK

    Message

    Parser

    Virtual

    Director

    Node

    Pass

    Receive


    Implementation9

    Implementation

    • Touch-event Processing

      • Check

        • ‘Down’: Test if hit some node

          • By SELECTION mode in OpenGL

          • If yes: add to TargetTable (event id → node)

        • ‘Move’, ‘Up’: Check if there is a target

          • Hit TargetTable by event id

          • ‘Up’: Remove from TargetTable

      • Dispatch

        • Dispatch events to each node

    i-m-Top SDK

    Message

    Parser

    Virtual

    Director

    Node

    Dispatch

    Check


    Implementation10

    Implementation

    • Touch-event Processing

      • Define event handler

        • Node decides how to act

        • Default handler

          • Move/Scale/Rotate under ‘Move’ events

        • Redefine handler

    • InitScenes( ){

    • //……

    • //add new event handlers to the photo object

  • photo.MultiFingerEvent += newMultiFingerHandler(MyMultiFingerHandler);

  • photo.FingerClick += newFingerClickHandler(MyFingerClickHandler);

  • }

  • i-m-Top SDK

    Message

    Parser

    Virtual

    Director

    Node

    Handle


    Implementation11

    Implementation

    P3

    P3

    P1

    P2

    P4

    P4

    Translation: T

    Scaling: S

    P3

    P3

    P1

    P34

    Rotation: R

    P12

    P2

    P4

    P4


    Implementation12

    Implementation

    • Reusable components

      • Multimedia

        • Video/audio

        • Camera capture

      • UI components

        • Virtual keyboard

        • Text-pad

        • Button, scrollbar…


    Outline5

    Outline

    • Introduction

    • Related Work

    • i-m-Top System

    • i-m-Top SDK

    • Application

    • Conclusion & Future Work


    Application

    Application

    • Map Browser

      • Different content in the fovea & peripheral

    • Photo Browser

      • Object/Region oriented fovea projection

    • Emotion Player

      • Interactive art

    c


    Outline6

    Outline

    • Introduction

    • Related Work

    • i-m-Top System

    • i-m-Top SDK

    • Application

    • Conclusion & Future Work


    Conclusion

    Conclusion

    • i-m-Top

      • Multi-resolution

        • Accommodating to human vision

        • Cost-effective large high-res display

      • Multi-touch

        • Intuitive interaction

    • i-m-Top SDK

      • Hide engineering challenges

        • Rapid prototyping applications

        • Investigating new interfaces / interactions


    Future work

    Future Work

    • Attentive computing

      • Combine with gaze-tracking

    • 3D interaction

      • Surface to space

    • Auto calibration

      • Vision-based

      • Continuous pan-tilt / homographymap


    A framework for multi resolution and multi touch systems

    Thanks for your listening


  • Login