1 / 29

Human Computation & ESP Game

Presenter: Lin, Sin-Yan. Human Computation & ESP Game. 2008/12/19. References. Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game , paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.

Download Presentation

Human Computation & ESP Game

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Presenter: Lin, Sin-Yan Human Computation & ESP Game 2008/12/19

  2. References • Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. • Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems. • Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4th international conference on Knowledge capture. • Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67. • Chen, L.-J., Wang, B.-C., Chen, C.-Y., King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC). • Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation.

  3. Outline • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments

  4. Human Computation • Concepts are started at 2001: The Open Mind Initiative • Some task are trivialfor humans, but continue to challenge computer algorithms • Treat human brains as processors in a distributed system, each can perform a small part of a massive computation • But humans require some incentive to become part of a collective computation • By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008) Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.

  5. Human Computation System- Peekaboom Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.

  6. Human Computation System- PhotoShoot (1/2) Cheng, K.-Y. (2006). PhotoShoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master.

  7. Human Computation System- PhotoShoot (2/2)

  8. Human Computation System- ImageHunter Chen, L.-H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62.

  9. Human Computation System- PhotoSlap Chang, T.-H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with PhotoSlap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52.

  10. Human Computation System- reCAPTCHA • Proposed by Ahn (2008) • Gathering the characters that optical character recognition (OCR) software has been unable to read

  11. Human Computation in E-Learning • 人力計算具有利用人類直覺的特性 • 動機 • 傳統英文閱讀學習及評量無法達成的地方 • 目的 • 用TAGs表達英文閱讀學習過程中最直覺的想法 • 方法 • Human Computation and Game Design

  12. Human Computation in E-Learning- why tag • Free, uncontrolled vocabulary • Social relations of TAGs image photo color design tagging art research

  13. Agenda • Human Computation • Human Computation System and Application • Human Computation in E-Learning • The ESP Game • Mathematical Model in ESP Game • System Gain • Image selection • Automatic playing the ESP Game • Problems of ESP Game • Conclusion and Comments

  14. The ESP Game • The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata Randomly selected image Input Keywords Input Keywords Keyword Match Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems

  15. Mathematic Model in ESP Game • There are many parameters in ESP Game • We can use those parameters to modeling many things that we want to know • Ex. • Number of players • Number of labels per game • Number of guesses to reach a consensus • Good and badword assumption • (Human Computation) System Gain • Image selection • User scoring model

  16. Mathematic Model in ESP Game- System Gain Modeling (1/2) • The performance of human computation system • The purpose of ESP Game tries to collect images’ tags • Two aspect of thinks • Take as many tags as possible for each image • Playing the maximum number of image • A metric to evaluate the system gain

  17. Mathematic Model in ESP Game- System Gain Modeling (2/2)

  18. Mathematic Model in ESP Game- Image Selection (1/3) • Different image selection method • Random • Fresh-first • Maximize ln(N) • Optimal (OPSA) • Largest possible system gain • P0: all images that have not been played • P1: all images that have been played at least once, but less than r times • P2: other images

  19. Mathematic Model in ESP Game- Image Selection (2/3) T / N

  20. Mathematic Model in ESP Game- Image Selection (3/3) M: Total number of images in the system Wang, B.-C., C.-W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica.

  21. Mathematic Model in ESP Game- Automatic Playing the ESP Game (1/4) • Given enough instances of labeled images as training data • Similar with Language model • Assign probability to the next label to be added • Plays the ESP Game without looking at the image • Limitation: at least one “off-limits” in the image Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

  22. Mathematic Model in ESP Game- Language Modeling (1/2) • Model the sequence of word usage of a certain language • Given a Language Model: M • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)

  23. Mathematic Model in ESP Game- Language Modeling (2/2) • P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this) • Markov assumption: • To give reasonable predictions, only the prior local context (the last few words) affects the next word • n-gram word model • Bi-gram: • P(I love this game) = P(I) P(love|I) P(this|love) P(game|this)

  24. Mathematic Model in ESP Game- Automatic Playing the ESP Game (2/4) • Given enough instances of labeled images as training data • At least one “off-limits” in the image • t: next label • T: off-limits set • P(“t is next label” | “set T already present”) Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

  25. Mathematic Model in ESP Game- Automatic Playing the ESP Game (3/4) • P(“t is next label” | “set T already present”) • Conditional independence assumption • The problem of zero probability Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

  26. Mathematic Model in ESP Game- Automatic Playing the ESP Game (4/4) • The problem of zero probability Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

  27. Mathematic Model in ESP Game- Others • User scoring model • More informative tags

  28. Problems of ESP Game • There is a lot of redundancy in the tag sets • Even when tags are not exactly synonyms, they are often “to be expected” given the other tags • There is a tendency to match on colors • People tend to add more generic labels Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

  29. Conclusion and Comments • The first thing of Human Computing research is finding a clear PURPOSE • How your game or method to achieve the purpose

More Related