crowdsourcing n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Crowdsourcing PowerPoint Presentation
Download Presentation
Crowdsourcing

Loading in 2 Seconds...

play fullscreen
1 / 15

Crowdsourcing - PowerPoint PPT Presentation


  • 190 Views
  • Uploaded on

Crowdsourcing. Nathan McFarland Isaac Nichols February 26 th , 2013. Agenda. Crowdsourcing History and Background Crowdsourcing Demo Worker Quality Control Theories and Tactics Crowdsourcing War Stories. Isaac’s Background.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Crowdsourcing


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
crowdsourcing

Crowdsourcing

Nathan McFarland

Isaac Nichols

February 26th, 2013

agenda
Agenda

Crowdsourcing History and Background

Crowdsourcing Demo

Worker Quality Control Theories and Tactics

Crowdsourcing War Stories

isaac s background
Isaac’s Background

Former UW/CSE Student. Worked for Microsoft & then Amazon after graduation.

Responsible for launching Amazon’s Mechanical Turk on Nov 2005

Sold crowdsourcing consulting business, TurkForce, to CrowdFlower in 2010

Founded Chirpsy in 2011…

nathan s background
Nathan’s Background

Started Career by working for 7 startups in 6 years, doing all manner code.

Followed that with a stint at Los Almos National Labs - doing a bunch of more academic things.

Founded CastingWords in Nov 2005 right when MTurk launched, ~7 year ago.

what is crowdsourcing
What is crowdsourcing?

Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”

— Jeff Howe 2006

what is crowdsourcing1
What is crowdsourcing?

Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”

— Jeff Howe 2006

what is crowdsourcing2
What is crowdsourcing?

Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”

— Jeff Howe 2006

what is crowdsourcing3
What is crowdsourcing?

Crowdsourcing is …“Engaging distributed groups of people with an open call to complete a task.”

— Jeff Howe 2006

crowd labor
Crowd Labor

Definition: Leveraging of a distributed virtual labor pool, available on-demand to fulfill a range of tasks from simple to complex

Simple Micro-task Crowdsourcing

Complex Task Crowdsourcing

worker quality control
Worker Quality Control

Plurality / Majority Rules

Gold Standard Questions / Known Answers

Peer Review / Grader Validation Workflows

Algorithmic Approaches (Machine Learning)

Worker Reputation

Submission Credibility (Dynamic Quality Control)

crowdsourcing war stories
Crowdsourcing War Stories
  • The Good…
    • Workers can be extremely helpful and proactive
  • The Bad...
    • Mistakes can have long term consequences
  • The Ugly…
    • Unpredictable Fraud
questions
Questions?

isaacn@chirpsy.com

nathan@castingwords.com