1 / 35

Anne Madden Journal Club 27 th April 2015

Thelwall , Mike, et al. " Do altmetrics work? Twitter and ten other social web services. " PloS one 8.5 (2013): e64841. Anne Madden Journal Club 27 th April 2015. What are altmetrics ? How are they calculated? Paper: Rationale and methodology Results?

remmert
Download Presentation

Anne Madden Journal Club 27 th April 2015

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Thelwall, Mike, et al. "Do altmetrics work? Twitter and ten other social web services." PloS one 8.5 (2013): e64841. Anne Madden Journal Club 27th April 2015

  2. What are altmetrics? • How are they calculated? • Paper: Rationale and methodology • Results? • Discussion: do altmetrics “work”?

  3. Priem (2010): definition I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I'm liking #altmetrics. (tweeted at 3.28am on 29th September 2010) Followed by: http://altmetrics.org/manifesto/

  4. “No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped.” From the altmetrics manifesto. (Priem 2010)

  5. Look not only at number of mentions but also importance of sources http://support.altmetric.com/knowledgebase/articles/83337-how-is-the-altmetric-score-calculated

  6. Rationale for this study “To what extent do the altmetric indicators associate with citation counts?” Building some form of evidence base to underpin altmetrics?

  7. If you can prove that the red box relates to the green box, you can then make reasonable predictions with regard to later citation scores.

  8. Methodology • PubMed articles with a non-zero altmetricscore • Sign test: “..each article is compared only against the two articles published immediately before and after it….” • Self-citations removed – why? • Altmetric scores gathered between July 2011 – 1st Jan 2013 only articles of approximately the same age, which are similarly exposed to the same citation delay and usage uptake biases, are compared to each other. Authors are unlikely to have first heard of their own paper on social media – looking for causal relationship

  9. Definition of success “..if the altmetric score is higher than the average altmetric score of the two adjacent articles and its citation score is higher than the average of the two adjacent articles OR the altmetric score is lower than the average altmetric score of the two adjacent articles and its citation score is lower than the average of the two adjacent articles.”

  10. Results • Total no. of successes v no. of failures: overall success • No. of journals where success exceeds failure: success • Individual altmetric level: mixed results. Four altmetrics “significantly and positively correlate with citations..the correlation for Twitter is significant and negative”. • “..results provide strong evidence that six of the eleven altmetrics.. associate with citation counts. ..correlation effect size is unknown” Again exclusions as a very wide range of journals was included, some with too low a sample to identify statistical differences Non-zero in at least one altmetric, not in all of them. Uptake of Twitter growing exponentially, tweeting right up to end Dec 2012.

  11. Experiment. Starting from non-zero citations: a very rough comparison • PubMed papers, with citations • Timeframe: mid to end 2013 • Search: “Influenza” [Title] • 20 papers selected at random • 14 had non-zero citations

  12. Most cited also had highest altmetric score. 13 out of 14 with non-zero citations also had non-zero altmetrics

  13. Lowest altmetric score was for non-English paper (Chinese) Cut-off point where association between altmetric score and citation count lose significance.

  14. Boon 2014, IATUL Proceedings: Altmetrics is an Indication of Quality Research or Just HOT Topicshttp://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=2034&context=iatul

  15. Discussion • Anonymity in most altmetric tools • Multiple accounts: the “Scholarly Selfie” • “Gaming” – could affect both altmetric and citation scores • Relative labour-intensity and engagement with paper

  16. It’s this easy to get an altmetric score. No need to even have read the paper.

  17. GAMING

  18. Kudos provides a platform for researchers to explain and share their work for wider audiences, and to measure the effect of these actions on a range of metrics including full text usage, altmetrics, and citations. basic service is free for researchers to use; publishers and institutions pay a fee for access to support tools, information on article and author performance within Kudoshttp://www.jisc-collections.ac.uk/UKSG/344/Kudos/?n=121b8a3f-e721-4bc0-8354-7ad4d47f99de

  19. Groups doing it for themselves “Early experience with JACR tweet chats demonstrates that organizing Twitter microblogging activities around topics of general interest to their target readership bears the potential for medical journals to increase their audiences and reach.” Hawkins, C. Matthew, et al. "The Impact of Social Media on Readership of a Peer-Reviewed Medical Journal." Journal of the American College of Radiology 11.11 (2014): 1038-1043.

  20. Topical, timely, authoritative, clear message, relevance to a broad audience Appeals to a broader audience; the ready-made headline… The relevance of colours in altmetrics Other tips, mainly relating to Twitter, suggest best times to tweet and using humour or shock effects 4 Wikipedia pages (aubergine bit)

  21. Do altmetrics work? “Not everything that counts can be counted, and not everything that can be counted counts”

  22. Neither altmetrics nor citation scores are evidence of quality.

  23. Look at 2nd most popular article in terms of altmetric score. Unfortunately, popularity due to failure of peer-review, proof-reading and editing systems. Absolutely nothing to do with the intrinsic value of the paper. (See next slide for details)

  24. “Although association preferences documented in our study theoretically could be a consequence of either mating or shoaling preferences in the different female groups investigated (should we cite the crappy Gabor paper here?), shoaling preferences are unlikely drivers of the documented patterns ……”

  25. Citation Index not always an indication of quality either

  26. From Ben Goldacre on YouTube Publication bias Publishers chasing the papers that will boost JIFs, citation scores etc.

  27. Hopewell, Sally, et al. "Publication bias in clinical trials due to statistical significance or direction of trial results." The Cochrane Library (2009). “Trials with positive findings are more likely to be published and published quicker than trials with negative findings.” Failure to publish: “scientific misconduct” valid expectation on the part of trial participants that their participation will contribute to knowledge.

  28. “…impact of scholarly material, with an emphasis on social media outlets…” (Priem 2014) “..provide an insight into things we have not measured before….” Tattersall 2015 “…analyzing the impact of outputs in different formats.. as opposed to the analysis of only journal papers.” (Costas 2014) “born digital” material …impacts on diverse audiences including scholars but also practitioners, clinicians, educators and the general public. (Piwowar 2013) …more timely data, showing evidence of impact in days instead of years …which scholarly products are read, discussed, saved and recommended as well as cited. ..the impact of web-native scholarly products Altmetrics can include “born digital” material

  29. Scholarly journals or scholarly communication? “For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that according to the NSF, a scientist’s worth is not dependent solely on publications.” Piwowar, Heather. "Altmetrics: Value all research products." Nature 493.7431 (2013): 159-159.

  30. Djuricich, Alexander M., and Janine E. Zee-Cheng. "Live tweeting in medicine:'Tweeting the meeting'." International Review of Psychiatry 0 (2015): 1-7.

  31. Cochran, Amalia, et al. "Use of Twitter to document the 2013 Academic Surgical Congress." Journal of Surgical Research 190.1 (2014): 36-40.

  32. Aslam, Anoshé A., et al. "The Reliability of Tweets as a Supplementary Method of Seasonal Influenza Surveillance." Journal of medical Internet research 16.11 (2014). “….this study demonstrated increased accuracy in using Twitter as a supplementary surveillance tool for influenzaas better filtering and classification methods yielded higher correlations for the 2013-2014 influenza season than those found for tweets in the previous influenza season, where emergency department ILI rates were better correlated to tweets than sentinel-provided ILI rates.”

  33. H-indexoutdated? Register for your ORCID Identifier (http://orcid.org). For more info see: http://libguides.ucd.ie/orcid • Create an author profile on Google Scholar Citations (http://scholar.google.com/citations) • Create an impactstory account for your biog. (http://impactstory.org/) From: Rathemacher, Andrée J. "Article-Level Metrics and Altmetrics: New Ways to Measure the Impact of Your Research." (2014). • When publishing ask for permission to store a copy in the Institutional Repository (http://www.lenus.ie)

  34. “Do altmetrics work?” Has Thelwall proven his case? • Altmetrics were never intended to replace / anticipate / mimic citation counts. • Altmetrics have a far wider scope, with more focus on unpublished born-digital material and other audiences. Therefore, to prove they “work”, more multi-faceted research required.

More Related