1 / 26

Crowdsourcing with Amazon Mechanical Turk

Crowdsourcing with Amazon Mechanical Turk. Alex Quinn and Tom Yeh HCIL Symposium May 25, 2011. Interface options. Easy web-based tool http://requester.mturk.com REST and SOAP APIs (HTTP) Wrapper SDKs: Java, .NET, Perl, PHP, Ruby Command line tool TurKit (JavaScript) Boto (Python)

tasha-arnold
Download Presentation

Crowdsourcing with Amazon Mechanical Turk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crowdsourcing withAmazon Mechanical Turk Alex Quinn and Tom Yeh HCIL Symposium May 25, 2011

  2. Interface options • Easy web-based tool • http://requester.mturk.com • REST and SOAP APIs (HTTP) • Wrapper SDKs: Java, .NET, Perl, PHP, Ruby • Command line tool • TurKit (JavaScript) • Boto (Python) • CrowdLib (Python)

  3. Posting HITs – external Your URL in an IFRAME Your form must route results to Amazon. <form method="post" action="https://mturk.com/...">

  4. Posting HITs – standard QuestionForm XML

  5. Standard vs. External HITs Standard External Requires your own web server Content: anything! Answers: Your CGI form fields Inner scrolling (potentially) More flexible Easier to do fancy things • Amazon hosts everything • Content: HTML, Flash, applets • Answers: Standard controls • No JavaScript / CSS • More secure • Easier to get started

  6. API Documentation

  7. Question XML <QuestionFormxmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question> </QuestionForm>

  8. Example - TurKit <QuestionFormxmlns="..."> <Question> <QuestionIdentifier>food_name</QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text>What food is Maine known for?</Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer></FreeTextAnswer> </AnswerSpecification> </Question> </QuestionForm> varparams = { title : "Find food for a state", desc : "Given a US state find a food it is known for", question : questionXML, reward : 0.01, maxAssignments : 3 }; var hit = mturk.createHIT(params); hit = mturk.getHIT(hit,true); for( var j=0; j<hit.assignments.length; j++ ) { varfoodName = hit.assignments[j].answer.food_name; print(stateName + " : " + foodName); }

  9. Example - CrowdLib import crowdlib as cl, crowdlib_settings hit_type = cl.create_hit_type("Find food for a state", "Given a US state find a food it is known for", 0.01) fields = [cl.text_field("What food is Maine known for?")] hit = hit_type.create_hit(fields, max_assignments=3)

  10. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results

  11. Example – Command Line • Many little files! • state_foods.question  question XML • state_foods.input  parameters (i.e. states) • state_foods.properties  title, description, etc. • state_foods_start.bat  calls loadHITs.bat • state_foods_get_results.bat  fetch results • state_foods_review_results.bat  approve/reject • Creates even more… • state_foods.success  contains the HIT IDs • state_foods.results  all result details

  12. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results <QuestionForm xmlns="..."> <Question> <QuestionIdentifier> food_name </QuestionIdentifier> <IsRequired>true</IsRequired> <QuestionContent> <Text> What food is Maine known for? </Text> </QuestionContent> <AnswerSpecification> <FreeTextAnswer/> </AnswerSpecification> </Question> </QuestionForm>

  13. Example – Command Line state Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine ... • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results

  14. Example – Command Line title:Find food for a state description:Given a US state find a food it is known for keywords:culture, food, search reward:0.02 assignments:1 annotation:${state} # time limit: 30 minutes assignmentduration:1800 # expires in: 3 days hitlifetime:259200 # auto-pay after: 2 days autoapprovaldelay:1296000 • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results

  15. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results pushd bin call loadHITs %1 %2 %3 %4 %5 ^ -label ..\sf\state_foods ^ -input ..\sf\state_foods.input ^ -question ..\sf\state_foods.question ^ -properties ..\sf\state_foods.properties popd

  16. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results pushd ..\bin call getResults %1 %2 %3 %4 %5 ^ -successfile ..\sf\state_foods.success -outputfile ..\sf\state_foods.results popd

  17. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results pushd ..\..\bin call reviewResults %1 %2 %3 %4 %5 ^ -resultsfile ..\sf\state_foods.results popd

  18. Example – Command Line • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results hitid hittypeid 11MFAR2I9GUBPPX3H302KECWDGG45C 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1N2WE274D6AYYDS5F6ENYRH1NR7JAO 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1KBVYNQOT8Y85Z3T3H6YTS1CV3596T 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1NPDB0OPE4D7JGXHYWSMAPKKCVAELG 1MQHAMAJ67VR562077Y5NLM6WNOIO8 1HNBWMEOVNP9NL2O6SYS8WM19JGNIZ 1MQHAMAJ67VR562077Y5NLM6WNOIO8

  19. Example – Command Line "hitid" "hittypeid" "title" "description" "keywords" "reward" "creationtime" "assignments" "numavailable" "numpending" "numcomplete" "hitstatus" "reviewstatus" "annotation" "assignmentduration" "autoapprovaldelay" "hitlifetime" "viewhit" "assignmentid" "workerid" "assignmentstatus" "autoapprovaltime" "assignmentaccepttime" "assignmentsubmittime" "assignmentapprovaltime" "assignmentrejecttime" "deadline" "feedback" "reject" "Answer.answer" "11MFAR2I9GUBPPX3H302KECWDGG45C" "1MQHAMAJ67VR562077Y5NLM6WNOIO8" "Find food for a state" "Given a US state find a food it is known for" "US culture, food, search" "$0.02" "Mon Dec 06 21:50:25 EST 2010" "1" "0" "0" "0" "Reviewable" "NotReviewed" "Maine" "3600" "1296000" "Thu Dec 09 21:50:25 EST 2010" "http://requester.mturk.com/mturk/manage HIT?HITId=11MFAR2I9GUBPPX3H302KECWDGG45C" "1SPKY563G67J32JCIGFCFF0CVAKGF0" "A1JYO3377GTS1O" "Submitted" "Tue Dec 21 21:52:56 EST 2010" "Mon Dec 06 21:51:26 EST 2010" "Mon Dec 06 21:52:56 EST 2010" "" "" "" "" "" "lobster" • Many little files! • state_foods.question • state_foods.input • state_foods.properties • state_foods_start.bat • state_foods_get_results.bat • state_foods_review_results.bat • Creates even more… • state_foods.success • state_foods.results

  20. Reviewing assignments • Approve • Reject • Grant bonus Word to wise: Preview in Excel!

  21. Worker issues • Approval rating • Blocked by multiple requestors  banned!!! • Too many rejections  banned (???) • My rule: If they made a reasonable attempt to follow the directions, then pay. • If quality is bad, get in touch.

  22. Worker issues • Indiscriminate rejections…

  23. Qualification Requirements • Qualification Type ~ HIT Type • Qualification Requirement ~ HIT • Qualification ~ Assignment • Custom • Built-in • Granted

  24. Informed Consent

  25. Ways to receive results • Ad-hoc script • Polling • Notification receptor

  26. Terminology review • HIT Type • HIT • Assignment  (HIT, worker) • Answer  part of Assignment • Qualification Type • Qualification Requirement • Qualification (QR, worker) HIT Type HIT

More Related