human computer interaction query by sketch l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Human Computer Interaction Query by Sketch PowerPoint Presentation
Download Presentation
Human Computer Interaction Query by Sketch

Loading in 2 Seconds...

play fullscreen
1 / 18

Human Computer Interaction Query by Sketch - PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on

Human Computer Interaction Query by Sketch. Chi-Ren Shyu. Department of Computer Engineering and Computer Science University of Missouri-Columbia Columbia, MO 65211, U.S.A. Query by Sketch -- an HCI topic?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Human Computer Interaction Query by Sketch' - paul


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
human computer interaction query by sketch
Human Computer InteractionQuery by Sketch

Chi-Ren Shyu

Department of Computer Engineering and Computer Science

University of Missouri-Columbia

Columbia, MO 65211, U.S.A.

query by sketch an hci topic
Query by Sketch -- an HCI topic?
  • Human side: query behavior, retrieval result evaluation, training of semantics and database indexing.
  • Computer side: query processing from human’s sketches, feature extraction from sketch objects, database search mechanisms, etc.
some applications and systems
Some Applications and Systems
  • Photo (natural scene images) IBM

QBIC http://wwwqbic.almaden.ibm.com/

  • GIS – Egenhofer at U. of Maine

Spatial Query by sketch http://www.spatial.maine.edu/~max/RL.html

  • Medical Image – University of Missouri – Colmbia

WebHIQS

http://diglib1.cecs.missouri.edu:3243/FinalProj/index1.html

(experimental website)

qbs in diagnostic image databases
QBS in Diagnostic Image Databases
  • Diagnoses often involve findings related to spatial relationship among lesions and landmarks.
  • Pathologies could be detected based on many visual patterns that are “sketchable”, such as shapes of tumors, distribution of nodules, ect.

Query: “Retrieve database images that have similar lesion-landmark location with the sketched query (image)”

query methods in diagnostic image dbs
Query methods in Diagnostic Image DBS
  • Sketch on a blank sheet – users are provided with a GUI for drawing organs, landmarks, and lesions.
  • Sketch on an organ template – users are provided with a template of an organ and a GUI for drawing landmarks and lesions.
  • Sketch on an existing image – users are provided with a real diagnostic image and a GUI for drawing landmarks and lesions.
research opportunity feature extraction from sketches
Research opportunity – Feature Extraction from Sketches
  • Shapes from sketch objects, texture from sketched regions, gray scale from sketched vicinity, etc. Domain expertise will help to design computer vision algorithms for this purpose.
  • Forming a feature vector for image retrieval.
  • Searching a high-dimensional database for retrieving the most similar sketches?
research opportunity spatial modeling from sketches
Research opportunity – Spatial Modeling from Sketches

Anatomical landmarks in HRCT lung:

  • Lung regions (Automatic Extraction)
  • Fissures (Human in the loop)
  • Lobe partitions (HIL/AE)

Lesions vs. landmarks:

  • Interior
  • Adjacent
  • Across
research opportunity spatial modeling from sketches 2
Research opportunity – Spatial Modeling from Sketches - 2

Query lesion

Sketched by

the user

Why these three lesions should be retrieved?

spatial modeling from sketches 3

Degree of Truth for (inner-Adjacent, To-the-right)

(0.34, 0.00)

(0.15, 0.00)

(0.55, 0.31)

(1.00, 0.82)

(1.00, 0.96)

(0.83, 0.84)

(0.21, 0.99)

(0.00, 1.00)

Spatial Modeling from Sketches - 3

What about multiple landmarks with multiple lesions?

research opportunity semantic query from sketches
Research opportunity – Semantic Query from Sketches

Semantic or so-called ontology is very subjective and user-oriented. Not good for “general purposed” database retrieval. GREAT for Sketches!

The hope is: Sketches might be able to bring more desirable retrieval results from user’s aspects.

semantic query from sketches 3
Semantic Query from Sketches - 3

Sketch bronchial structures on an organ template and fill

gray scales to different part of the object, such as bronchial

Walls and lumens.

semantic query from sketches 4
Semantic Query from Sketches - 4

Users sketch on many training images

and form a semantic tree structure

for future retrieval.

semantic query from sketches 5
Semantic Query from Sketches - 5

For fuzzy gangs, this looks familiar. We can form

membership functions for the semantic

terms by using user’s Sketches.

semantic query from sketches 6
Semantic Query from Sketches - 6

Ranking database images by semantics

research opportunity query languages for sketches
Research opportunity – Query Languages for Sketches

SELECT I.iid

FROM LungImageDatabase I

WHERE SIM( SKETCH(Qid,GUI), I.FE(iid)) < threshold

ORDER BY SIM;

A possible SQL extension for QBS:

summary
Summary

Query by sketch is not new, however, it has high potential for many applications, especially in the areas of diagnostic image database, GIS, criminal investigation, etc.

In the CECS department, we have many existing research projects that are highly correlated to the concept of QBS. We should have some sort of synergistic efforts to have some interesting projects in the near future.