Methodologies for measuring research impact for a new interdisciplinary institute

1. Methodologies for measuring research impact for a new interdisciplinary institute Janet Baker and Dr Nina Prasolova Queensland University of Technology Library

2. Complicated subject and so best told as a story • The request • The task – impact report for a “virtual” institute • The challenge • The negotiations • The methodologies • The workflow • The lessons we learned

3. Characteristics of Institutes or Research Groups Established institutes • Clear purpose and identity • Set agenda, coherent workflow • Membership list • Physical space with known teams • High quality resources already available to measure research impact • Benchmarking progressive and peer group known

4. Institute for Future Environments “Virtual” institute • Based around broad concepts, a Big Idea or evolving research themes with loosely affiliated individual researchers • “Fluid” relationships • Disparate output types and citing cultures to interpret

5. Interdisciplinarity: a field of study that crosses traditional boundaries between academic disciplines, which integrates these different approaches or methods (Center for the Study of Interdisciplinarity). Boston Interdisciplinary Instituteshttp://www.bu.edu/cgs/citl/interdisciplinary-institutes/ Literature review reveals increasing number of reports and interest in a growing body of literature around issues, policies, challenges regarding IDR

6. Challenges • No clear boundaries or disciplinary structure • Loosely affiliatedclusters • Defining Interdisciplinarity - subject categories or research codes • Multiple variables – type of publication, where indexed, differing culture of citation, collaborative arrangements • Expectations – make clear what is possible

7. Expectations/ Negotiations: What is the goal? To evaluate performance – factors to consider Look at the traditional metrics – what are you trying to illustrate, what can you use - • Which indicators are consistent for group • What tools will give greatest coverage/accuracy • Is it possible to analyse cited references - interdisciplinarity • Journal subject categories? – WoS and Scopus • Percentiles and Average Citation Metrics • Percentage share of total publications

8. Expectations - What is the goal? To predict and for strategic planning • Co-authorship graphs – analysis of effective collaborations • Co-citation analysis (SciVal) identifying emerging trends • Identify Networks outside the discipline boundaries To improve communications and reporting efficiency • Find other research support staff also collecting and evaluating impact? • Tacit knowledge (people evaluating are not the researchers) • Knitting together support around the researcher

9. What is the reality? • Each subgroup is different – refine the theme and vocabulary • The type of research output dictates the assessment tools • Which metrics are available across the theme • Benchmarking strategy is specific for the subgroup/cluster

10. Identifying clusters – establish borders • Field of Research Codes • MeSH or other Thesauri • Journal Categories • Keyword analysis

11. Street computing : towards an integrated open data Application Programming Interface (API) for cities http://eprints.qut.edu.au/59160/

12. Analysis of productivity and impact • Percentage of the outputs in the theme • Distribution of citations (average and percentile) • Collaborations • Interdisciplinarity • Downloads • Attention Metrics • Media releases

13. Metrics of Research Productivity and Impact

14. ESI Table

15. Workflow

16. Scopus benchmarking (share and percentiles of ‘cited by’)“Tropical Crop*” (All fields) (limit to Country: Australia) QUT share in the total pubs: 33 (36%) out of 91 3 QUT papers in 10th percentile and 9 QUT papers in 25th percentile

18. Author collaborations

19. Analysing collaborations in SciVal SciVal (Elsevier) January, 2014

20. Summary: what we have learnt • For “virtual” institutes it is important to identify clusters • Baselines – sometimes need to change, cultures differ • Google Scholar Citations Profile – appears most useful for some clusters, but data needs to be checked • Possible to manually granulate by retrieving those publications that are in the theme or cluster and then benchmark your institute • Final report needs detailed explanation of metrics and tools chosen and their applicability and replicability

21. What we learned • There is a place for librarians in this exercise • For emerging new institutes librarians could be instrumental providing input into shaping the research clusters and showing future opportunities in interdisciplinarity and collaborations

22. References Huutoniemi, K., Klein, J. T., Bruun, H., & Hukkinen, J. (2010). Analyzing interdisciplinarity: Typology and indicators. Research Policy, 39(1), 79-88. doi: http://dx.doi.org/10.1016/j.respol.2009.09.011 Roessner, D., Porter, A. L., Nersessian, N. J., & Carley, S. (2013). Validating indicators of interdisciplinarity: linking bibliometric measures to studies of engineering research labs. Scientometrics, 94(2), 439-468. Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., . . . Börner, K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26. Rymer, L. (2011). Go8 Backgrounder 23 | Measuring the impact of research – The context for metric development. Turner, ACT: The Group of Eight. Retrieved from Center for the Study of Interdisciplinarity. (2013). Retrieved from http://csid.unt.edu/research/ria_overview.html LSE Public Policy Group. (2011). Maximizing the impacts of your research: A handbook for social scientists (Consultation Draft 3). Retrieved from http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf

23. Creative Solutions Valerie Ucumari(CC BY-NC-ND 2.0)