1 / 10

Evaluation Methods for Social Systems

Evaluation Methods for Social Systems. Joan DiMicco IBM Research Center for Social Software. IBM’s Center for Social Software. Cambridge, MA Within IBM Research social software Collaboration enterprise communication social visualization I manage the Visual Communication Lab.

kylynn-torres
Download Presentation

Evaluation Methods for Social Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation Methods for Social Systems Joan DiMicco IBM Research Center for Social Software

  2. IBM’s Center for Social Software • Cambridge, MA • Within IBM Research • social software • Collaboration • enterprise communication • social visualization • I manage the Visual Communication Lab

  3. Background • We build social communities • Measure success of these communities • Our challenge as researchers: quantifying essentially qualitative experiences • Overview of 5 different analysis approaches

  4. Social Software Community • We built a social network site for IBM employees • How do we get people to join and then continue to use? • Reputation, skills, your merit • Photos • Commenting / sharing • Incentive systems (e.g. points) and honorable roles

  5. Broad summaries from interview data • Asked of us: What is the ROI on social networking? • We asked: Why are people using our SNS site? • Focused subject interviews on this question (~20 ppl) • Coded each interview statement based on the type of motivation • Themes emerged: 1. social/personal connecting, 2. career climbing, and 3. spreading marketing message of pet projects • Then looked content on site for alignment with these themes JM DiMicco, DR Millen, W Geyer, C Dugan, B Brownholtz, M Muller. (2008) “Motivations for Social Networking at Work.” Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW 2008). `

  6. Using surveys to measure success • Question: What impact does our SNS have on IBM? (similar to ROI question) • We used “social capital” as a concept to measure how employees have more access to information and expertise • Through existing survey instruments, we measured social capital of employees. • Based on their reported behavior on the SNS, we correlated (regression analysis) behavior on site with greater social capital. C Steinfeld, JM DiMicco, N Ellison, C Lampe. (2009) “Bowling Online: Social Networking and Social Capital within the Organization. Fourth International Conference on Communities and Technologies (C&T 2009).

  7. Measuring an intervention • We deployed as new feature on the SNS (rewarding points for content). How do we measure impact? • Controlled the deployment: only half of people saw the feature • Compared the two groups • Fully instrumented the system to observe behavior R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Proceedings of CHI 2008.

  8. Using a survey to gather ground truth • Does your behavior on the SNS reflect your feelings about the people? • We designed a survey “game” asking “How strong is your relationship with these people?” • Used this ground truth to compare with behavior on site • Regression analysis demonstrated correlations between closeness and certain behaviors A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

  9. Using interviews to inform quantitative analysis • A website for viewing legislation: how are people using it? • Logs of which features were used and how long each user stayed • Determined few power users, many casual users • Interviewed 6 power users to answer our questions raised during our log analysis of “why?” A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

  10. Where from here? • What is your question to answer? • How can you measure it? • Can you measure it by proxy? • Do you have the data or do you need to collect? • Surveys • System logging • Interviews • Controlled study

More Related