1 / 26

Amazon Mechanical Turk

Crowdsourcing Quality Management and other stories Panos Ipeirotis New York University & Tagasauris. Amazon Mechanical Turk. Tagging. patent pending. But what if MTurk workers submit incorrect tags?. patent pending. Voting. Voting.

airell
Download Presentation

Amazon Mechanical Turk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CrowdsourcingQuality Managementand other storiesPanosIpeirotisNew York University & Tagasauris

  2. Amazon Mechanical Turk

  3. Tagging patent pending

  4. But what if MTurk workers submit incorrect tags? patent pending

  5. Voting

  6. Voting But what if MTurk workers also vote incorrectly?

  7. Voting But what if MTurk workers also vote incorrectly? Redundancy: 5+ votes per tag

  8. But my task is not just right or wrong… Describe this Creation HIT(e.g. describe the image) Compare HIT (voting) Which is better? Improve HIT (e.g. improve description)

  9. version 1: A parial view of a pocket calculator together with some coins and a pen. version 2: A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocketand wallet sized item used for business, writting, calculating prices or solving math problems and purchasing items. version 3: A close-up photograph of the following items: A CASIO multi-function calculator. A ball point pen, uncapped. Various coins, apparently European, both copper and gold. Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance. version 4: …Various British coins; two of £1 value, three of 20p value and one of 1p value. … version 8: “A close-up photograph of the following items: A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance."

  10. New York: Old and Newhttp://newyorkoldandnew.blogspot.com/ An engagement experiment: “Search the NYPL digital library for an old photo of New York, and then find a photo of the same location, today”

  11. 1935

  12. 2007

  13. 1933

  14. 2007

  15. 1890

  16. 2007

  17. 1890

  18. 1900

  19. 1976

  20. 2008

  21. The incomplete knowledge graph [Harsen Homestead, 72nd Street, ca. 1888.]

  22. The incomplete knowledge graph HarsenHomestead ???

  23. The incomplete knowledge graph Jacob Harsen, a farmer who started Harsenville, a village spanning 68th Street to 81st Street between Central Park West and the Hudson River. Source: http://www.oneshermansquare.com/HistoryofShermanSquare.aspx

  24. The incomplete knowledge graph Crowd not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]

  25. New York Times, 1911

  26. The incomplete knowledge graph Crowd participation not only annotates historical archives but also contributes back to the knowledge graph [allowing easier annotation in the future]

More Related