Quality Management on Amazon Mechanical Turk Panos Ipeirotis Foster Provost Jing Wang New York University. Example: Build an Adult Web Site Classifier. Need a large number of hand-labeled sites Get people to look at sites and classify them as:
G (general audience), or X (restricted/porn)
Our friend ATAMRO447HWJQmarked all sites as G.Seems like a spammer…
The Dawid & Skene algorithm generates “confusion matrixes” for workers
How to see if a worker is a spammer using the confusion matrix?
Small ambiguity for R votes but other than that, fine!
The results are highly ambiguous. No information provided!
Demo and Open source implementation available at: