1 / 31

Enhancing Human-Machine Communication via Visual Attributes

Enhancing Human-Machine Communication via Visual Attributes. Devi Parikh Virginia Tech. Interacting with Vision Systems. User. Supervisor. Interacting with Vision Systems. Mode of communication is important. Semantic Gap. Interacting with Vision Systems. Necessary for communication

ruana
Download Presentation

Enhancing Human-Machine Communication via Visual Attributes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Enhancing Human-Machine Communication via Visual Attributes Devi Parikh Virginia Tech

  2. Interacting with Vision Systems User Supervisor

  3. Interacting with Vision Systems Mode of communication is important Semantic Gap

  4. Interacting with Vision Systems • Necessary for communication • Language that humans understand (semantic) • Language that machines understand (visual) • Attributes • Example: furry, natural, chubby, shiny, etc. • Better features, deeper image understanding, etc. Farhadi et al., Kumar et al., Lampert et al., etc. • Human-machine communication

  5. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  6. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  7. Image Search Query: “black shoes” … Binary Relevance Feedback

  8. Image Search Query: “black shoes” … “more formalthan these” “shinierthan these” …

  9. Relative Attributes Linear ranking function: open Training Testing Openness [Parikh and Grauman, ICCV 2011]

  10. Image Search • System has pre-trained relative attribute predictors • Relevance of image = # constraints satisfied … “shinier” “more formal”

  11. WhittleSearch shiny … “shinier” “more formal” formal

  12. WhittleSearch shiny formal

  13. WhittleSearch [Kovashka, Parikh and Grauman, CVPR 2012] (Patent pending) 13

  14. Whittle Search: Demo (Online) [Prepared by NamanAgrawal, Demo at CVPR 2013] (Patent pending) 14

  15. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  16. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  17. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  18. Traditional Active Learning Is this a forest? No, this is not a forest.

  19. Classifier Feedback I think this is a forest. What do you think ? No, this is too opento be a forest. [Images more open than query] Ah! These images must not be forests either then. [Parkashand Parikh, ECCV 2012] …

  20. Classifier Feedback I think this is a forest. What do you think ? No, this is too opento be a forest. Pre-trained relative attributes [Images more open than query] Ah! These images must not be forests either then. …

  21. Classifier Feedback I think this is a forest. What do you think ? No, this is too opento be a forest. Learn attributes on the fly [Images more open than query] Ah! These images must not be forests either then. …

  22. Classifier Feedback I think this is a forest. What do you think ? No, this is too opento be a forest. [images labeled as forest] Ah! These images must be less open than query …

  23. Classifier Feedback • Learning attributes on the fly • Start only with unlabeled images (+ a supervisor) • Categories and attributes learnt from scratch • Confidence in instances • Active learning for learning with attributes-based classifier feedback [Biswasand Parikh, CVPR 2013]

  24. Classifier Feedback Accuracy Parkash and Parikh ECCV 2012 Biswas and Parikh CVPR 2013 Number of iterations

  25. User Role of the Human Supervisor Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  26. WhittleSearch Query: “black shoes” … “more formalthan these” “shinierthan these” …

  27. Image Search [Parikh andGrauman, ICCV 2013]

  28. Saying the Right Thing • Improved image search, description Not smiling Smiling more than [Sadovnik, Gallagher, Parikh and Chen, ICCV 2013]

  29. Saliency of Attributes • Improved image search, zero-shot learning, description Scary, sharp teeth White, furry [Turakhia and Parikh, ICCV 2013]

  30. User Role of the Human Supervisor Integrating AI with today’s machine learning tools Image Search Instilling Domain Knowledge Human Polar bears are white and larger than rabbits. My missing brother is fuller-faced than this boy. Accessing user’s intensions for mental image search Getting more from what the human says without added human effort Enhanced human-machine communication via attributes for improved visual recognition Active and Interactive Learning Supervisor User Communicator Characterizing Failure Modes Interpretable Models Trustworthy systems: keyfor effective human-machine teams More usable computer vision systems even with their imperfections I think this is a polar bear because this is a white and furry animal. If the image is blurry or the face is not frontal, I may fail. Reading Between the Lines Machine User Supervisor

  31. Thank you!

More Related