1 / 15

Crowdsourcing

Crowdsourcing. Nathan McFarland Isaac Nichols February 26 th , 2013. Agenda. Crowdsourcing History and Background Crowdsourcing Demo Worker Quality Control Theories and Tactics Crowdsourcing War Stories. Isaac’s Background.

rhoda-potter
Download Presentation

Crowdsourcing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crowdsourcing Nathan McFarland Isaac Nichols February 26th, 2013

  2. Agenda Crowdsourcing History and Background Crowdsourcing Demo Worker Quality Control Theories and Tactics Crowdsourcing War Stories

  3. Isaac’s Background Former UW/CSE Student. Worked for Microsoft & then Amazon after graduation. Responsible for launching Amazon’s Mechanical Turk on Nov 2005 Sold crowdsourcing consulting business, TurkForce, to CrowdFlower in 2010 Founded Chirpsy in 2011…

  4. Nathan’s Background Started Career by working for 7 startups in 6 years, doing all manner code. Followed that with a stint at Los Almos National Labs - doing a bunch of more academic things. Founded CastingWords in Nov 2005 right when MTurk launched, ~7 year ago.

  5. What is crowdsourcing? Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

  6. What is crowdsourcing? Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

  7. What is crowdsourcing? Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

  8. What is crowdsourcing? Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.” — Jeff Howe 2006

  9. Crowd Labor Definition: Leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex Simple Micro-task Crowdsourcing Complex Task Crowdsourcing

  10. History Lesson:The Original Mechanical Turk

  11. 5 Minute Mechanical Turk Demo…

  12. Worker Quality Control Plurality / Majority Rules Gold Standard Questions / Known Answers Peer Review / Grader Validation Workflows Algorithmic Approaches (Machine Learning) Worker Reputation Submission Credibility (Dynamic Quality Control)

  13. Crowdsourcing War Stories • The Good… • Workers can be extremely helpful and proactive • The Bad... • Mistakes can have long term consequences • The Ugly… • Unpredictable Fraud

  14. Questions? isaacn@chirpsy.com nathan@castingwords.com

More Related