1 / 55

The Multi-Dimensional Research Assessment Matrix

The Multi-Dimensional Research Assessment Matrix. Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012. Short CV Henk F. Moed. Contents. (o) Bibliometrics makes sense.

ghenriksen
Download Presentation

The Multi-Dimensional Research Assessment Matrix

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

  2. Short CV Henk F. Moed

  3. Contents

  4. (o)Bibliometrics makes sense

  5. Which country has these main collaborators? USA

  6. Which country has these main collaborators? Brazil

  7. Which country has these main collaborators? Malaysia

  8. Which country has these main collaborators? Romania

  9. Which country has these main collaborators? SouthAfrica

  10. Contents

  11. (i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

  12. (ii) Bibliometric tools help establishing a longer term perspective in research funding

  13. (iii) One must be cautious using “societal benefit” as an assessment criterion of basic research: it can not be measured in a politically neutral way

  14. (iv) Citations measurescientific-scholarly impact rather than quality or validity

  15. (v) Citation counts in social sciences and humanities may be influenced by politicalideologies

  16. Citation impact and ideology Fall of the Berlin wall in Nov. 1989

  17. (vi)The future of research assessment lies in the intelligent combination of metrics and peer review

  18. (vii)Case study on funding policies of a National Research Council reveals biases in peer review

  19. Affinity Applicants – Evaluation Committee 0 Applicants are/were not member of any Committee • Co-applicant is/was member of a Committee, but notof the one evaluating • Firstapplicant is/was member of a Committee, but not of the one evaluating • Co-applicant is member of the Committee(s) evaluating the proposal • Firstapplicant is member of the Committee(s) evaluating the proposal

  20. For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4) % SUBMITTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

  21. Probability to be granted increases with increasing affinity applicants-Committee % GRANTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

  22. Logistic regression analysis:Affinity Applicant-Committee has a significant effect upon the probability to be granted MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Reltransdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

  23. (viii)Data must be accurate and verifiable

  24. (ix)Assessed researchers must have the opportunity to check data and comment on outcomes

  25. (x)A framework is needed to characterize and position bibliometric indicators and products

  26. Contents

  27. The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

  28. Multi‐dimensional Research Assessment Matrix (Part)

  29. Multi‐dimensional Research Assessment Matrix (Part) Read column- wise

  30. Multi‐dimensional Research Assessment Matrix (Part)

  31. Multi‐dimensional Research Assessment Matrix (Part)

  32. MD-RAM: Example 1 • Publications in international jrnls; • Actual citation impact • Individual • Hiring/promotion • Productivity & impact • PhD date, place, supervisor; • Invitations for conferences

  33. Multi‐dimensional Research Assessment Matrix (Part)

  34. MD-RAM: Example 2 • Publications in international jrnls; • (Trend in) actual citation impact • Research group • Monitoring research program • Scientific impact • Collaborations • Topicality

  35. Research assessment methodologies: Important considerations • Methodology must befit-for-purpose • What is the primary “problem” to besolved? • Be aware of unintendedeffects • Change a methodologyevery 5-10 years • What is anacceptable “errorrate”? • Wrong inindividualcases  benificiaryfor the system as a whole

  36. Contents

  37. Journal articles + citations Journal full text data Journal usage data Unit of assess- ment Books Patents Conference Procs Trade jrnls Newspapers Social media

  38. (a)Downloads vs. CitationsWhat do full article downloads measure?

  39. Authors vs. readers Readers Authors ?

  40. Hypothesis on degree of correlation between downloads and citations Authors Authors Readers Readers Strong Weak

  41. Usage vs. citations per main field Scientific ? PSYCHOL Societal

  42. (b)Patent citations to journal articles:The technological impact of research

  43. The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012] PATENTS (TotalPatent) Citations by patent examiners and inventors 42 LIBRARY SCIENCE JOURNALS (Scopus)

  44. Cited articles: keywords in titles The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development Citing patents: keywords in titles The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

  45. (c) International scientific migration

  46. International migration vs. co-authorship

  47. Map of countries with Ratio migration/collaboration > 1.2

  48. Languagesimilarity drives migrationstrongerthanit drives co-authorship Politicaltensions affect migrationlessthanthey affect co-authorship

  49. (d)Citation context analysisCombining citation data from Scopus with full text article data from ScienceDirect

  50. The use of contextual citations analysis to disclose the thematic and conceptual flow of cross- disciplinary research:the case of the Journal of Informetics 2007 (Gali Halevi et al., to be published, 2012)

More Related