1 / 48

23. April 2013

Introduction to Collaborative Web . Sergej Zerr zerr@L3S.de. 23. April 2013. 1. Outline. Collaborative Advantages Wisdom of crowds Conditions for a successful collaboration Using Collaborative Data Gathering Data from Social Web / Mechanical Turk Inter Rater Agreement

sylvia
Download Presentation

23. April 2013

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to Collaborative Web . Sergej Zerr zerr@L3S.de 23. April 2013 1

  2. Outline • Collaborative Advantages • Wisdom of crowds • Conditions for a successful collaboration • Using Collaborative Data • Gathering Data from Social Web / Mechanical Turk • Inter Rater Agreement • Search and Sensemaking processes, the overview • Collaboration opportunities in (Web) search • Overview Papers • Collaborative Search, Search result relevance judgments,.. • Small experiment • Can we collaborate? • Discussion Sergej Zerr 2

  3. Collaboration Oftenweneedmorethanonehand Sometimesmorethanonebrain “Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations” James Suroewicki Sergej Zerr 3

  4. Direct Collaboration in WWW can be used: Collaborative tagging, Favorite assignments, Click logs, Data gathering, Reccomendations, ect., ect.,ect…. Tags: Rainbow, Sea, Island, Green, Palmtree, Maui Sergej Zerr 4

  5. Social Computer Expert Masses Equivalent, or greater, utility under the curve Utility # ofcontributors 10 10,000+ Sergej Zerr 5

  6. Social Computer Expert $$$$ Masses $ Utility 100,000 amateurs 1.6 Million articles 5 years to develop Real-Time Updates 4,000 experts 80,000 articles 200 years to develop Annual Updates # ofcontributors 10 10,000+ Sergej Zerr 6

  7. Collaboration: Paradox Collaborative workneedstobemanaged efficiently http://spectrum.ieee.org/computing/hardware/multicore-is-bad-news-for-supercomputers Kasparovwonagainsttheworld Sergej Zerr 7

  8. Collaboration Sergej Zerr 8

  9. http://en.wikipedia.org/wiki/Groupthink Groupthink Symptoms: Irving Lester Janis (26 May 1918 - 15 November 1990) • Direct pressure on dissenters • Self-censorship • Illusion of unanimity • Self-appointed ‘mindguards’ • Illusion of invulnerability • Collective rationalization • Belief in inherent morality • Stereotyped views of out-groups Sergej Zerr 9

  10. Collaboration “The best collective decisions are the product of disagreement and contest, not consensus or compromise.” “The best way for a group to be smart is for each person in it to think and act as independently as possible.” Sergej Zerr 10

  11. Many Complementary Games • ESP Game: label images  Image retrieval by text • Squigl: match the labels to areas  Object recognition • Matchin: find the better image  Image ranking • FlipIt: memory with similar images • Near duplicate detection • Other areas covered as well: label songs, find synonyms, describe videos • See: www.gwap.com by Luis von Ahn Sergej Zerr 11

  12. Re-Using Human Power Sergej Zerr 12

  13. Outline • Collaborative Advantages • Wisdom of crowds • Conditions for a successful collaboration • Using Collaborative Data • Gathering Data from Social Web / Mechanical Turk • Inter Rater Agreement • Search and Sensemaking processes, the overview • Collaboration opportunities in (Web) search • Overview Papers • Collaborative Search, Search result relevance judgments,.. • Small experiment • Can we collaborate? • Discussion Two different images that share the same ESP labels: man and woman Sergej Zerr 13

  14. Social Computer • Human can (yet) solve some tasks • more efficient and/or accurate • as a machine would do. • Captcha • Classification (OCR) • Image tagging • Speech recognition Sergej Zerr 14

  15. Social Computer – Good Question „Mr. Burns saw Homer withthebinoculars“ Matthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourcing-for-search-evaluation-and-socialalgorithmic-search Sergej Zerr 15

  16. Social Computer – Good Interface • Throw the coin and tell us the result • Head? • Tail? • Results • Head 61 • Tail 39 • People often tend just to select the first option  • Better: Some preliminary textual answer • Coin type? • Head or tail. Matthew Lease and Omar Alonso: http://de.slideshare.net/mattlease/crowdsourcing-for-search-evaluation-and-socialalgorithmic-search Sergej Zerr 16

  17. L3S Research – “What about image privacy?” Sergej Zerr , Stefan Siersdorfer , Jonathon Hare , Elena Demidova Privacy-Aware Image Classificationand Search , SIGIR‘12 Sergej Zerr 17

  18. L3S Research • Gathering average community notion of privacy • We crawled “most recently uploaded” Flickr photos (2 Months) • Started a social annotation game (over the course of 2 weeks) • 81 users (colleagues, social networks , forum users) , 6 teams Sergej Zerr , Stefan Siersdorfer , Jonathon Hare , Elena Demidova Privacy-Aware Image Classificationand Search , SIGIR‘12 Sergej Zerr 18

  19. Sampling Data from Social Web: Tags 1-20 tags per photo >75 tags per photo Sergej Zerr 19

  20. Sampling Data from Social Web: User activity Simple random sample canresult in a setdominatedbyfew power user Sergej Zerr 20

  21. Inter–Rater Reliability: What about ambiguity? Nederland, netherlands, holland, dutch Rotterdam, wielrennen, cycling, duck le grand depart, tour de france, Reklame, caravan, Funny Fotos Sergej Zerr 21

  22. Inter–Rater Reliability “Where is the cat?” V V X V V X X X X X X X X X V V X V V V X X X X Results Sergej Zerr 22

  23. Inter–Rater Reliability “Where is the cat?” • Ideas • Introduce „Honeypot“ – a setofgroundtruthobjectsandselectonlygoodraters • Select onlyraterswho rate closetoaveragerating. • Other startegies? V V X V V X X X X X X X X X V V X V V V X X X X Results Sergej Zerr 23

  24. Inter–Rater Reliability: A B Naive approach: 3 cases out of 6 = 0.5 agreement Kilem L. Gwet, Handbook of inter-raterreliability 2010 Sergej Zerr 24

  25. Inter–Rater Reliability: Cohen’s Kappa(1960) Idea: Weneedtoremoveagreementachived just bychance Kilem L. Gwet, Handbook of inter-raterreliability 2010 Sergej Zerr 25

  26. Inter–Rater Reliability: Missing Values Idea: Use partial ratingstoestimate marginal probabilityonly Kilem L. Gwet, Handbook of inter-raterreliability 2010 Sergej Zerr 26

  27. Inter–Rater Reliability: Extentions • Multiple Raters/Categories: • Fleiss 1971 – Averageoverrandompairsofratersforrandomobjects • AdjustmentforOrdinalandInterval Data, Weighting: • weightjudgementsusingdistancesbetweencategories. • ReduceKappa Paradox: • Split judgementsintomorecertain / marginal. • Measures: AC1, AC2(ordinalandintervaldata) • Check forstatisticalsignificance: • The numberofcategoriesand/orratersmatters. Kilem L. Gwet, Handbook of inter-raterreliability 2010 Sergej Zerr 27

  28. Inter–Rater Reliability: Kappa Interpretations Pleasenote: These interpretationswereproventobeusefullmostly in medicaldomain(diagnosis) Kilem L. Gwet, Handbook of inter-raterreliability 2010 Sergej Zerr 28

  29. Useful human power for annotating the Web • 5000 people playing simultaneously can label all images on Google in 30 days! • Individual games in Yahoo! and MSN average over 5,000 players at a time • Possible contributions: attach labels to images in other languages, categorize web pages into topics Sergej Zerr 29

  30. Outline • Collaborative Advantages • Wisdom of crowds • Conditions for a successful collaboration • Using Collaborative Data • Gathering Data from Social Web / Mechanical Turk • Inter Rater Agreement • Search and Sensemaking processes, the overview • Collaboration opportunities in (Web) search • Overview Papers • Collaborative Search, Search result relevance judgments,.. • Small experiment • Can we collaborate? • Discussion Sergej Zerr 30

  31. What is relevant? Sergej Zerr 31

  32. Search and Sensemaking Process Query Annotation/Organization Resultlist Can collaborationimprovethesensemakingprocessatanystep? Do youusecollaborativesearch? Sergej Zerr 32

  33. What are the typical collaborative search tasks? • Watched over someone’s shoulder as he/she searched the Web, and suggested alternate query terms. • E-mailed someone links to share the results of a Web search. • E-mailed someone a textual summary to share the results of a Web search. • Called someone on the phone to tell them about the results of a Web search. Morris, M.R. A Survey of Collaborative Web Search Practices. In Proceedings of 26th CHI Conference 2008 Around 90% of Microsoft employees are engaged in collaborative search activities. Morris, M.R. A Survey of Collaborative Web Search Practices. In Proceedings of 26th CHI Conference 2008 Sergej Zerr 33

  34. Collaborative Search: • Query formulation • Search process/browsing • Save/Bookmark • Annotate/Organize • Howtosupportusers in collaborativesearching? • Ideas • Tools(Web 2.0) Sergej Zerr 34

  35. Indirect Collaboration on Web: Google uses search/click logs. For PageRank algorithm each link to a page serves as a vote for that page. Amazon uses search/click logs. For item recommendation similar users are the indirect voters for the product. ect., ect. Sergej Zerr 35

  36. Outline • Collaborative Advantages • Wisdom of crowds • Conditions for a successful collaboration • Using Collaborative Data • Gathering Data from Social Web / Mechanical Turk • Inter Rater Agreement • Search and Sensemaking processes, the overview • Collaboration opportunities in (Web) search • Overview Papers • Collaborative Search, Search result relevance judgments,.. • Small experiment • Can we collaborate? • Discussion Sergej Zerr 36

  37. Software support for Co-located search (CoSearch). Amershi, S., Morris, M. CoSearch: System for colocated collaborative Web search. In Proceedings of 26th CHI Conference 2008 Sergej Zerr 37

  38. Software support for Co-located search (CoSearch). Amershi, S., Morris, M. CoSearch: System for colocated collaborative Web search. In Proceedings of 26th CHI Conference 2008 Sergej Zerr 38

  39. Spatial distributed Search (SearchTogether) • (a) integrating messaging • (b) query awareness, • (c) current results • (d) recommendation queue • (e)(f)(g) search buttons • (h) page-specific metadata • (i) toolbar • (j) browser Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Collaborative Web Search. In Proceedings of the UIST 2007 Sergej Zerr 39

  40. Spatial distributed Search (SearchTogether) • 38% af all result lists were the consequence of using history • 70 positive and 9 negative ratings • 22 of 36 recommendation were viewed by the recipients Morris, M.R. & Horvitz, E. SearchTogether: An Interface for Collaborative Web Search. In Proceedings of the UIST 2007 Sergej Zerr 40

  41. Improvement Through Collaborative Ranking Agrahri, A., Manickam, D., Riedl, J. Can people collaborate to improve the relevance of search results? In Proceedings of the ACM International Conference on Recommendation Systems 2008 Sergej Zerr 41

  42. WeSearch: Collaborative Sensemaking (Hardware support) Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference on Computer supported cooperative work (CSCW '10). ACM, New York, NY, USA, 401-410. Sergej Zerr 42

  43. WeSearch: Collaborative Sensemaking Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference on Computer supported cooperative work (CSCW '10). ACM, New York, NY, USA, 401-410. Sergej Zerr 43

  44. Hardware support for Co-located(TeamSearch). • Circles are categories: people, location, year, event Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Comparing Techniques for Co-Present Collaborative Search of Digital Media. Sergej Zerr 44

  45. Hardware support for Co-located(TeamSearch). Morris, M.R., Paepcke, A., and Winograd, T. Team-Search: Comparing Techniques for Co-Present Collaborative Search of Digital Media. Sergej Zerr 45

  46. Outline • Collaborative Advantages • Wisdom of crowds • Conditions for a successful collaboration • Using Collaborative Data • Gathering Data from Social Web / Mechanical Turk • Inter Rater Agreement • Search and Sensemaking processes, the overview • Collaboration opportunities in (Web) search • Overview Papers • Collaborative Search, Search result relevance judgments,.. • Small experiment • Can we collaborate? • Discussion Sergej Zerr 46

  47. Sergej Zerr ?

  48. Smalltalk, what should be the properties of a perfect collaborative system? Sergej Zerr 48

More Related