Video based document tracking unifying your physical and electronic desktops
Download
1 / 95

Video-Based Document Tracking: - PowerPoint PPT Presentation


  • 254 Views
  • Updated On :

Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops. Jiwon Kim Steven M. Seitz Maneesh Agrawala University of Washington Microsoft Research. Motivation. Unifying physical and electronic desktops. Video camera. Record video of paper on physical desktop. Desktop.

Related searches for Video-Based Document Tracking:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Video-Based Document Tracking:' - Mia_John


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Video based document tracking unifying your physical and electronic desktops l.jpg

Video-Based Document Tracking:Unifying Your Physical and Electronic Desktops

Jiwon Kim Steven M. Seitz Maneesh Agrawala

University of Washington Microsoft Research



Unifying physical and electronic desktops l.jpg
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

Desktop


Unifying physical and electronic desktops4 l.jpg
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking

Desktop


Unifying physical and electronic desktops5 l.jpg
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking

  • Recognition

Desktop


Unifying physical and electronic desktops6 l.jpg
Unifying physical andelectronic desktops

Video camera

Record video of paper on physical desktop

  • Tracking

  • Recognition

  • Linking

Desktop


Applications l.jpg
Applications

Video camera

Desktop


Applications8 l.jpg
Applications

Video camera

  • Find lost document

Desktop


Applications9 l.jpg
Applications

Video camera

  • Find lost document

  • Browse remote desk

Desktop


Applications10 l.jpg
Applications

Video camera

  • Find lost document

  • Browse remote desk

  • Find electronic version

Desktop


Applications11 l.jpg
Applications

Video camera

  • Find lost document

  • Browse remote desk

  • Find electronic version

  • History-based queries

Desktop




Related work l.jpg
Related Work

  • Interactive desktops

DigitalDesk [Wellner 93]


Related work15 l.jpg
Related Work

  • Interactive desktops

Self-Organizing Desk

[Rus et al. 97]

DigitalDesk [Wellner 93]


Related work16 l.jpg
Related Work

  • Interactive desktops

  • Augmentedpaper

PADD [Guimbretière 03]


Related work17 l.jpg
Related Work

  • Interactive desktops

  • Augmentedpaper

CyberCode

[Rekimoto et al. 00]

PADD [Guimbretière 03]


Related work18 l.jpg
Related Work

  • Interactive desktops

  • Alternative media

  • Object tracking & recognition

SIFT [Lowe 04]


System overview l.jpg
System Overview

Video camera

Computer

User

Desk


System overview20 l.jpg
System Overview

Video of desk


System overview21 l.jpg
System Overview

Images from PDF

Video of desk


System overview22 l.jpg
System Overview

Images from PDF

Video of desk

Track & recognize


System overview23 l.jpg

Desk

Desk

System Overview

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1


System overview24 l.jpg

Desk

Desk

System Overview

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1

Scene Graph


System overview25 l.jpg

Desk

Desk

System Overview

Where is my W-2?

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1


System overview26 l.jpg
System Overview

Where is my W-2?

Answer

Internal representation

Images from PDF

Video of desk

Track & recognize

Desk

Desk

T

T+1


System overview27 l.jpg

Desk

Desk

System Overview

Where is my W-2?

Internal representation

Images from PDF

Video of desk

Track & recognize

T

T+1



Tracking recognition29 l.jpg

Tracking & Recognition

Event


Event types l.jpg
Event Types

before

after

Move


Event types31 l.jpg
Event Types

before

after

Move

Entry


Event types32 l.jpg
Event Types

before

after

Move

Entry

Exit


Tracking recognition33 l.jpg

Tracking & Recognition

Event

Desk


Tracking recognition34 l.jpg

Tracking & Recognition

Event

Desk

Desk


Tracking recognition35 l.jpg

Tracking & Recognition

Event

sanders01.pdf

lowe04sift.pdf

tut-article.pdf

objectspaces.pdf

kidd94.pdf

Desk

Desk


Assumptions l.jpg
Assumptions

  • Document

    • Corresponding electronic copy exists

    • No duplicates of same document


Assumptions37 l.jpg
Assumptions

  • Document

    • Corresponding electronic copy exists

    • No duplicates of same document

  • Motion

    • 3 event types: move/entry/exit

    • One document at a time

    • Only topmost document can move


Non assumptions l.jpg
Non-Assumptions

  • Desk need not be initially empty


Non assumptions39 l.jpg
Non-Assumptions

  • Desk need not be initially empty

  • Stacks may overlap


Algorithm overview l.jpg
Algorithm Overview

Input Frames


Algorithm overview41 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after


Algorithm overview42 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”


Algorithm overview43 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf


Algorithm overview44 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk


Algorithm overview45 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk


Event detection l.jpg

Event Detection


Event detection47 l.jpg
Event Detection

Frame differences

time


Event detection48 l.jpg
Event Detection

Frame differences

time


Event detection49 l.jpg
Event Detection

Frame differences

time


Event detection50 l.jpg
Event Detection

Frame differences

time


Event detection51 l.jpg

Event Detection

Frame differences

time


Event detection52 l.jpg

Event Detection

Motion Frames

Event Frames

Threshold

Image motion

time

time


Event detection53 l.jpg

Event Detection

Motion Frames

before

after


Algorithm overview54 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk


Event interpretation l.jpg
Event Interpretation

before

after

Move

Entry

Exit


Event interpretation56 l.jpg
Event Interpretation

before

after

Move

1. Move vs. Entry/Exit

Entry

Exit


Event interpretation57 l.jpg
Event Interpretation

before

after

Move

Entry

2. Entry vs. Exit

Exit


Event interpretation58 l.jpg
Event Interpretation

before

after

Move

1. Move vs. Entry/Exit

Entry

Exit


Move vs entry exit l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit60 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit61 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit62 l.jpg
Move vs. Entry/Exit

after

before


Event interpretation63 l.jpg
Event Interpretation

  • Use SIFT[Lowe 04]

    • Rotation- and scale-invariant

    • Highly distinctive (128-bit vector)


Move vs entry exit64 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit65 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit66 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit67 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit68 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit69 l.jpg
Move vs. Entry/Exit

after

before


Move vs entry exit70 l.jpg
Move vs. Entry/Exit

Motion: (x,y,θ)

after

before


Algorithm overview71 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk


Document recognition l.jpg
Document Recognition

  • Match against PDF image database

File2.pdf

File3.pdf

File4.pdf

File5.pdf

File6.pdf

File1.pdf


Document recognition73 l.jpg
Document Recognition

  • Performance analysis

    • Tested 20 pages against database of 162 pages


Document recognition74 l.jpg
Document Recognition

  • Performance analysis

    • Tested 20 pages against database of 162 pages

    • ~200x300 pixels per document for reliable match

Recognition Rate

Document Resolution


Document recognition75 l.jpg
Document Recognition

  • Performance analysis

    • Tested 20 pages against database of 162 pages

    • ~200x300 pixels per document for reliable match

0.9

Recognition Rate

300

Document Resolution


Algorithm overview76 l.jpg
Algorithm Overview

Input Frames

Event Detection

before

after

Event Interpretation

“A document moved from (x1,y1) to (x2,y2)”

File1.pdf

Document Recognition

File2.pdf

File3.pdf

Scene Graph Update

Desk

Desk


Scene graph update l.jpg
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk


Scene graph update78 l.jpg
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk


Scene graph update79 l.jpg
Scene Graph Update

Motion: (x,y,θ)

after

before

Desk

Desk


Results l.jpg
Results

  • Input video

    • ~40 minutes

    • 1024x768 @ 15 fps

    • 22 documents, 49 events

  • Running time

    • Video processed offline

    • No optimization

    • A few hours for entire video






Future work l.jpg
Future Work

  • Enhance realism

  • More applications


Future work86 l.jpg
Future Work

  • Enhance realism

    • Handle more realistic desktops




Future work89 l.jpg
Future Work

  • Enhance realism

    • Handle more realistic desktops

    • Real-time performance


Future work90 l.jpg
Future Work

  • More applications

    • Support other document tasks

      • E.g., attach reminder, cluster documents


Future work91 l.jpg
Future Work

  • More applications

    • Support other document tasks

      • E.g., attach reminder, cluster documents

    • Beyond documents


Future work92 l.jpg
Future Work

  • More applications

    • Support other document tasks

      • E.g., attach reminder, cluster documents

    • Beyond documents


Future work93 l.jpg
Future Work

  • More applications

    • Support other document tasks

      • E.g., attach reminder, cluster documents

    • Beyond documents


Future work94 l.jpg
Future Work

  • More applications

    • Support other document tasks

      • E.g., attach reminder, cluster documents

    • Beyond documents


Acknowledgments l.jpg
Acknowledgments

  • NSF

  • Intel Corp.

  • Li Zhang


ad