video based document tracking unifying your physical and electronic desktops
Download
Skip this Video
Download Presentation
Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops

Loading in 2 Seconds...

play fullscreen
1 / 95

Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops - PowerPoint PPT Presentation


  • 255 Views
  • Uploaded on

Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops. Jiwon Kim Steven M. Seitz Maneesh Agrawala University of Washington Microsoft Research. Motivation. Unifying physical and electronic desktops. Video camera. Record video of paper on physical desktop. Desktop.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops' - Mia_John


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
video based document tracking unifying your physical and electronic desktops

Video-Based Document Tracking:Unifying Your Physical and Electronic Desktops

Jiwon Kim Steven M. Seitz Maneesh Agrawala

University of Washington Microsoft Research

unifying physical and electronic desktops
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

Desktop

unifying physical and electronic desktops4
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking

Desktop

unifying physical and electronic desktops5
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking
  • Recognition

Desktop

unifying physical and electronic desktops6
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking
  • Recognition
  • Linking

Desktop

applications
Applications

Video camera

Desktop

applications8
Applications

Video camera

  • Find lost document

Desktop

applications9
Applications

Video camera

  • Find lost document
  • Browse remote desk

Desktop

applications10
Applications

Video camera

  • Find lost document
  • Browse remote desk
  • Find electronic version

Desktop

applications11
Applications

Video camera

  • Find lost document
  • Browse remote desk
  • Find electronic version
  • History-based queries

Desktop

related work
Related Work
  • Interactive desktops

DigitalDesk [Wellner 93]

related work15
Related Work
  • Interactive desktops

Self-Organizing Desk

[Rus et al. 97]

DigitalDesk [Wellner 93]

related work16
Related Work
  • Interactive desktops
  • Augmentedpaper

PADD [Guimbretière 03]

related work17
Related Work
  • Interactive desktops
  • Augmentedpaper

CyberCode

[Rekimoto et al. 00]

PADD [Guimbretière 03]

related work18
Related Work
  • Interactive desktops
  • Alternative media
  • Object tracking & recognition

SIFT [Lowe 04]

system overview
System Overview

Video camera

Computer

User

Desk

system overview20
System Overview

Video of desk

system overview21
System Overview

Images from PDF

Video of desk

system overview22
System Overview

Images from PDF

Video of desk

Track & recognize

system overview23
Desk

Desk

System Overview

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1

system overview24
Desk

Desk

System Overview

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1

Scene Graph

system overview25
Desk

Desk

System Overview

Where is my W-2?

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1

system overview26
System Overview

Where is my W-2?

Answer

Internal representation

Images from PDF

Video of desk

Track & recognize

Desk

Desk

T

T+1

system overview27
Desk

Desk

System Overview

Where is my W-2?

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1

event types
Event Types

before

after

Move

event types31
Event Types

before

after

Move

Entry

event types32
Event Types

before

after

Move

Entry

Exit

tracking recognition34

Tracking & Recognition

Event

Desk

Desk

tracking recognition35

Tracking & Recognition

Event

sanders01.pdf

lowe04sift.pdf

tut-article.pdf

objectspaces.pdf

kidd94.pdf

Desk

Desk

assumptions
Assumptions
  • Document
    • Corresponding electronic copy exists
    • No duplicates of same document
assumptions37
Assumptions
  • Document
    • Corresponding electronic copy exists
    • No duplicates of same document
  • Motion
    • 3 event types: move/entry/exit
    • One document at a time
    • Only topmost document can move
non assumptions
Non-Assumptions
  • Desk need not be initially empty
non assumptions39
Non-Assumptions
  • Desk need not be initially empty
  • Stacks may overlap
algorithm overview
Algorithm Overview

Input Frames

algorithm overview41
Algorithm Overview

Input Frames

Event Detection

before

after

algorithm overview42
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

algorithm overview43
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

algorithm overview44
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk

algorithm overview45
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk

event detection47
Event Detection

Frame differences

time

event detection48
Event Detection

Frame differences

time

event detection49
Event Detection

Frame differences

time

event detection50
Event Detection

Frame differences

time

event detection51

Event Detection

Frame differences

time

event detection52

Event Detection

Motion Frames

Event Frames

Threshold

Image motion

time

time

event detection53

Event Detection

Motion Frames

before

after

algorithm overview54
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk

event interpretation
Event Interpretation

before

after

Move

Entry

Exit

event interpretation56
Event Interpretation

before

after

Move

1. Move vs. Entry/Exit

Entry

Exit

event interpretation57
Event Interpretation

before

after

Move

Entry

2. Entry vs. Exit

Exit

event interpretation58
Event Interpretation

before

after

Move

1. Move vs. Entry/Exit

Entry

Exit

event interpretation63
Event Interpretation
  • Use SIFT[Lowe 04]
    • Rotation- and scale-invariant
    • Highly distinctive (128-bit vector)
move vs entry exit70
Move vs. Entry/Exit

Motion: (x,y,θ)

after

before

algorithm overview71
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk

document recognition
Document Recognition
  • Match against PDF image database

File2.pdf

File3.pdf

File4.pdf

File5.pdf

File6.pdf

File1.pdf

document recognition73
Document Recognition
  • Performance analysis
    • Tested 20 pages against database of 162 pages
document recognition74
Document Recognition
  • Performance analysis
    • Tested 20 pages against database of 162 pages
    • ~200x300 pixels per document for reliable match

Recognition Rate

Document Resolution

document recognition75
Document Recognition
  • Performance analysis
    • Tested 20 pages against database of 162 pages
    • ~200x300 pixels per document for reliable match

0.9

Recognition Rate

300

Document Resolution

algorithm overview76
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk

scene graph update
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk

scene graph update78
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk

scene graph update79
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk

Desk

results
Results
  • Input video
    • ~40 minutes
    • 1024x768 @ 15 fps
    • 22 documents, 49 events
  • Running time
    • Video processed offline
    • No optimization
    • A few hours for entire video
future work
Future Work
  • Enhance realism
  • More applications
future work86
Future Work
  • Enhance realism
    • Handle more realistic desktops
future work89
Future Work
  • Enhance realism
    • Handle more realistic desktops
    • Real-time performance
future work90
Future Work
  • More applications
    • Support other document tasks
      • E.g., attach reminder, cluster documents
future work91
Future Work
  • More applications
    • Support other document tasks
      • E.g., attach reminder, cluster documents
    • Beyond documents
future work92
Future Work
  • More applications
    • Support other document tasks
      • E.g., attach reminder, cluster documents
    • Beyond documents
future work93
Future Work
  • More applications
    • Support other document tasks
      • E.g., attach reminder, cluster documents
    • Beyond documents
future work94
Future Work
  • More applications
    • Support other document tasks
      • E.g., attach reminder, cluster documents
    • Beyond documents
acknowledgments
Acknowledgments
  • NSF
  • Intel Corp.
  • Li Zhang
ad