1 / 30

Context Assistance in Media Interpretations

Context Assistance in Media Interpretations. Yohan Jin and B. Prabhakaran Department of Computer Science University of Texas at Dallas, Richardson, TX 75083 praba@utdallas.edu http://multimedia.utdallas.edu. Context Assistance. “Approximating Graph Algorithm”. Semantic Dimensional

fonda
Download Presentation

Context Assistance in Media Interpretations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Context Assistance in Media Interpretations Yohan Jin and B. Prabhakaran Department of Computer Science University of Texas at Dallas, Richardson, TX 75083 praba@utdallas.edu http://multimedia.utdallas.edu

  2. Context Assistance “Approximating Graph Algorithm” Semantic Dimensional Reduction 3D Motion Capture Data Video Human Motion Recognition “3D mocap in HMM” Automatic Image Annotation Semantic Distance Measure WordNet, Web (Google) Image Annotation Refinement Web-Image Annotation Refinement

  3. Motivation Query by CBIR Retrieved image • There is a gap between perceptual issue and conceptual issue. • Semantic gap: Hard to represent semantic meaning using low-level image features like color, texture and shape. • It’s possible to answer query ‘Red ball’ with ‘Red Rose’.

  4. Motivation Contd. • Semantic Grouping fish sea choose coral Relevant keywords building flower remove

  5. WordNet • Car: a motor vehicle with four wheels; usually propelled by an internal combustion engine .self-propelled vehicle . wheeled vehicle . vehicle .instrumentation . artifact .object .entity

  6. Leacock and Chodorow Measure Outdoor game golf Field game Ball game Professional golf Baseball game ShortestLen (professional golf, baseball game) = 5 D=17(overall depth)

  7. Resnik Measure Determine the lcs (lowest common subsumer) between two words (hotel,door) IC value of lcs is the semantic similarity value Introduces first Information Content (IC) Use the Corpus (SemCor2.0) A concept with high IC value Concept has a more detailed information IC(“Cable-television”) greater than IC(“television”) artifact structure building door hotel

  8. Web? There is answer..but Woods, hoping to extend winning streak, charges to lead at Dubai Desert Classic Published: January 30, 2008 DUBAI, United Arab Emirates (AP) — Tiger Woods picked up right where he left off last week - at the top of the leaderboard. Woods, who won the Buick Invitational on Sunday by eightstrokes, shot a 7-under 65 Thursday to take a two-shot lead after the first round of the Dubai Desert Classic. "I played well today, just a bunch of good golf shots,"" Woods said after his bogey-free round at the EmiratesGolf Club. Elevenplayers, including Miguel Angel Jimenez and Abu Dhabi Golf Championship winner Martin Kaymer, were tied for second at 67. Ernie Els, Sergio Garcia and defending champion Henrik Stenson were tied with 10 others another stroke back. Woods said he played better in Dubai than he did last week at Torrey Pines. "I had two good days of practice the last couple days and started to hit the ball a lot better than I did last week,"" said Woods, who won the Dubai tournament in 2006. “woods” “Tiger Woods” “golf shot” “stroke”

  9. Problem Reduction building woods desktop sky G=(V,E) fish V politics E Woods, Building, fish, Sky, desktop, politics “Semantic Distance” “Candidate keywords” “Image Annotation Refinement”  weighted Maximum Cut Problem

  10. Optimal Solution of Weighted Maximum-Cut (-1) building (-1) woods (-1) desktop sky (1) We need to check Possibilities. (1) fish politics (1)

  11. Randomized Approximation Scheme (2-way)

  12. Relaxation Effect on Image Annotation Refinement building building crystal anemone crystal anemone palace reef palace reef people people Edge-Values Node-Variable

  13. Image Annotation Refinement…

  14. 2-D Random-hyper Plane

  15. 2-dimensional Random-hyper plane for decision

  16. Result with Corel Image Set

  17. Another Semantic Distance (Normalized Google Distance) NGD: Find the Google PageCount when both words are used together in a search. E.g., “Speakers” and “sound” would have a relatively high number of result pages when compared to “speakers” and “elephant.”

  18. Result with Web-images

  19. Motion Capture System

  20. Spatial Dimensional Reduction

  21. Motivation 3D Motion Capture feature values closer to the “Semantic” data Video Human Gesture Recognition Problem is susceptible to noisy environment and human subject Design a method for combining these two heterogeneous data with Hidden Markov Model

  22. Video Human Motion Motion History Image based Low-level Feature Extraction

  23. Video and 3D mocap video 3D mocap. video 3D mocap. Time-series Distribution

  24. Vision Based Framework

  25. Vision Based Framework

  26. Experiments • Data Set • 1,321 video motion clips, 7 motion capture clips • 11 subjects • 7 motion classes • Comparison between Baum-Welch & KBH method

  27. Challenges • Variations among different subjects: ‘backhands’ action by 4 different persons

  28. Result1

  29. Result2 Improvement in each motion Classes KBH achieved more than 2 times better accuracy in the motion classes which showed bad “local” maxima (region ‘A’)

  30. Questions? • Combine sensor context and keyword context? • Effect of human in the loop: • If humans make changes to the max-cut solutions, how to interpret it? • Other sensors’ effect: such as GPS for gesture recognition in pervasive health care.

More Related