1 / 29

The Benefits and Challenges of Collecting Richer Object Annotations

The Benefits and Challenges of Collecting Richer Object Annotations. Ian Endres , Ali Farhadi , Derek Hoiem , David Forsyth University of Illinois at Urbana-Champaign. What should we say about objects?. What should we say about objects?. Mirrors. Vehicle Two-wheeled Motorcycle.

ehren
Download Presentation

The Benefits and Challenges of Collecting Richer Object Annotations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Benefits and Challenges of Collecting Richer Object Annotations Ian Endres, Ali Farhadi, Derek Hoiem, David Forsyth University of Illinois at Urbana-Champaign

  2. What should we say about objects?

  3. What should we say about objects? Mirrors Vehicle Two-wheeled Motorcycle Gas tank Seat Headlight Lic. Plate Motorcycle Facing right On the street Has a rider Tail light Metal Exhaust Rubber Engine Wheel Wheel

  4. How should we annotate objects? • Pascal VOC Motorcycle

  5. Object Polygons • LabelMe • Distinguish object pixels from background • Malisiewicz, Efros BMVC ‘07 Motorcycle

  6. Attributes Motorcycle Two-wheeled Vehicle • Rich descriptions • Recognize unfamiliar objects • Farhadi et al. CVPR’09 • Lampert et al. CVPR‘09 Facing right Has a rider On the street Seat Wheel Headlight Tail light Handlebars Side mirror Exhaust pipe Engine License plate Motorcycle Painted metal Bare metal Rubber Plastic

  7. Localized Parts • Improved models • Part configuration • Better pose/viewpoint • Cross-category part localization: A. Farhadi, I. Endres, D. Hoiem CVPR‘10 Mirrors Gas tank Seat Headlight Lic. Plate Motorcycle Tail light Exhaust Engine Wheel Wheel

  8. Materials Glass Bare Metal Painted Metal Rubber

  9. How should we annotate objects? Is motorcycle Is two-wheeled Is vehicle Facing right On street Has rider Has seat Has wheel Has headlight Has tail light Has handlebars Has side mirror Has exhaust pipe Has engine Has license plate Has painted metal Has bare metal Has rubber +

  10. The CORE datasetCross-category Object REcognition • 2780 Images – from ImageNet • 3192 Objects – 28 Categories • 26695 Parts – 71 types • 30046 Attributes – 34 types • 1052 Material Images – 10 types Download or browse online: http://vision.cs.uiuc.edu/CORE

  11. Outline of Dataset Creation Objects Attributes Images MTurk Quality Control CORE Parts Materials

  12. Collecting Data: Images Unusual Examples Stylized Images Foreground Only Canonical Poses

  13. Collecting Data: Images

  14. Amazon’s Mechanical Turk Turk Accept Submit Task Completed Task Dataset Reject + Resubmit

  15. Collecting Data: Binary Attributes

  16. Collecting Data: Polygons From Alex Sorokin: http://code.google.com/p/cv-web-annotation-toolkit/

  17. Collecting Data: Materials From Alex Sorokin: http://code.google.com/p/cv-web-annotation-toolkit/

  18. Quality Assurance:Grading Multiple annotations If annotations can be compared easily Visual Inspection If inspection is faster than annotation

  19. Quality Assurance:Establishing Trust Qualification Must qualify to complete tasks Higher barrier of entry Good work record Auto-accept all work after 10 tasks and 90% acceptance rate Automatically accept ~50%

  20. Collection Results: Binary Attributes Before 81.4% Agreement 1. Too many options 2. Unnatural tasks 3. Unclear interface 4. Ambiguous options

  21. Collection Results: Binary Attributes Ambiguous Viewpoint • Before • Facing to the right • Facing toward the camera • After • Right side visible • Front side visible

  22. Collection Results: Binary Attributes Before 81.4% Agreement After 90.7% Agreement

  23. Collection Results: Object Polygons • 5% rejection rate Excellent Acceptable Rejected

  24. Collection Results: Part Polygons • 25% rejection rate Excellent Acceptable Rejected

  25. Collection Results: Materials • 22% rejection rate Excellent Acceptable Rejected

  26. Ensuring High Throughput • Provide a large steady stream of work • Attracts more workers, keeps regulars • Quick grading turnaround • Don’t expect results overnight when using United States workers • Do expect results overnight when using global workers • Beware of language issues

  27. How Much Does it Cost?For 100 Images Turk workers spend ~25 hours Annotator Payment + ~10% Turk Commission

  28. Take Home Messages • We need detailed annotations for rich recognition • Mechanical Turk can provide detailed annotations • Interface/instruction design important • State succinctly, simply • Keep Task Size Manageable • Quality Control necessary

  29. Find and Describeunfamiliar objects. Animal Vehicle Four-legged Mammal Head Wheel Leg Can run, jump Is herbivorous Facing right Moves on road Facing right • Come see our poster on Wednesday: • “Attribute-Centric Recognition for Cross-Category Generalization”

More Related