1 / 37

Giorgio Sirilli ISSiRFA - CNR

Giorgio Sirilli ISSiRFA - CNR. Giorgio Sirilli ISSiRFA - CNR. Our subject: Evaluation. Knowledge: the bundle. Knowledge. There are many different types of knowledge and knowledge holders. Not all knowledge is scientific, much knowledge is practical and contex specific

beau
Download Presentation

Giorgio Sirilli ISSiRFA - CNR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Giorgio Sirilli ISSiRFA - CNR

  2. Giorgio Sirilli ISSiRFA - CNR

  3. Our subject: Evaluation

  4. Knowledge: the bundle

  5. Knowledge There are many different types of knowledge and knowledge holders. Not all knowledge is scientific, much knowledge is practical and contex specific Society is interested in its: Production Transmission Use

  6. Research evaluation: the ladder Knowledge New public management Evaluation Research Assesment Exercise (RAE-REF) Bibliometrics CNR Department SSH

  7. The learned society University teaching research “third mission” Researchagencies research problemsolving management

  8. New Public Management Catchwords Accountability Valueformoney

  9. “Performance basedresearchfundingsystems” The rationale “The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.” Herbst, 2007

  10. “Performance basedresearchfundingsystems” Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding

  11. Share of university funding dependent on “Performance-Based Research Funding Systems” Criteria Research (teaching excluded) Evaluation ex-post (ex-ante excluded) Research output Research funding must depend on the results of evaluation National system (internal university evaluations excluded)

  12. Share of university funding dependent on “Performance-Based Research Funding Systems” Hicks D., Reseach Policy (2012)

  13. Share of university funding dependent on “Performance-Based Research Funding Systems” “The distribution of university research funding is something of an illusion” “It is the competition for prestige that creates powerful incentives within university systems” “Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity” Hicks D., Reseach Policy (2012)

  14. Evaluation of CNR Institutes Carried out by CNR in 2009-2010 Evaluation of structures, not individuals Methodology • General panel • Panels (26) composed of 156 scientists • Documentation provieded by Institutes • Site visit • Final score Lessons: “The General Panel recommended that in the future the two groups of scientific areas (natural sciences and engineering, social science and humanities) remain separated. Experience showed in fact that differences in objectives, research methodologies, evaluation criteria suggest to be deal with them separately”

  15. ANVUR VQR (Evaluation of the Quality of Research) Habilitation of university professors bibliometric areas non-bibliometric areas Both exercises largely based on bibliometrics

  16. Knowledge: the bundle

  17. When two types of evaluation do not match

  18. Fourtypesofpublications Internationally oriented journal articles Books National journals Enlightenment literature Hicks D., Research Policy (2012)

  19. The fourliteratures International journal literature Books High impact Other means of transfer WoS/Scopus Enlightenment National literature Knowledge transfer Local context Hicks D., Research Policy (2012)

  20. Journal literature Problem: missing output Social Science Science Journal articles – counted in WoS/Scopus indicators 45-61% 81-85% The 3 other literatures – not counted Other: books, edited books, book chapters, monographs, reports, creative works Sources: Bourke et al,1996; Pestana et al., 1995; Winterhager, 1994

  21. Norwegian social science – WoScoveragebyfield Percentage of journal articles indexed in WoS Sciences Ossenblok et al. 2012 Ossenblok et al. 2012

  22. National literature The problem with national literature: Lower level of critique and peer review, questionable quality

  23. Enlightenmentliteraturebyfield Sciences Burnhill & Tubby-Hille, 1994, UK social science

  24. Research evaluation

  25. CNR Department of Cultural Identity and Heritage Cultural identity 15 Institutes and 336 researchers/technologists SSH Cultural heritage 4 Institutes and 110 researchers/technologists SSH + NSE

  26. CNR Departments - Researchers and technologists by training(percentage) 10% 10% 38% Cultural identity (15 Institutes) 42% 18% • Cultural heritage • (4 Institutes) 46% 33% 3% Significant differences in specialisation of researchers between former Departments

  27. CNR Departments – Research products (2007-2011)(percentage) Significant similarities in types of “products” between former Departments

  28. CNR Departments – Research products (2007-2011)(percentage) Structural difference between SSH Departments and a NSE Department

  29. CNR Departments – Journal articles (2008-2010) 36% 46% 11% 23% 53% 31% Structural difference between former Departments in terms of output

  30. Curatela • Curatela • Brevetto • Brevetto • Altra tipologia • Composizione • Performance • Banca dati • Software • Cartografia • Commento giuridico • Rapporti progetti di ricerca • Rapporti tecnici, manuali, carte geologiche e tematiche e prodotti multimediali • Rassegne, recensioni, prefazioni • Altre pubblicazioni • Rapporti finali progetti di ricerca • Materiale didattico • Rapporti di commissioni • Rapporti statistici • Note interne, manuali e guide • Prodotti multimediali • Risultati di valorizzazione applicativa • Progetti • Disegni e design • Rappresentazioni • Mostre ed esposizioni • Manufatti, prototipi d'arte e relativi progetti • Rapporti tecnici/preprint/workingpaper • Data set • Altro prodotto Contributo in rivista Articolo in rivista Traduzione in rivista Nota a sentenza Scheda bibliografica Abstract in rivista Abstract/Comunicazione in rivista Abstract/Poster in rivista Rassegna e recensione Contributo in volume Contributo in volume (capitolo o saggio) Traduzione in volume Prefazione/Postfazione Breve introduzione Voce (in dizionario o enciclopedia) Scheda di catalogo Contributo in atti di convegno Contributo in atti di convegno Abstract/Poster in atti di convegno Comunicazione a convegno Abstract/Poster in convegno Abstract/Comunicazione in atti di convegno Libro Monografia o trattato scientifico Traduzione di libro Edizione critica Commento scientifico Concordanza Indice Pubblicazione di fonti inedite Antologia e manuale scolastico Dizionario e thesaurus Bibliografia, repertorio, glossario Catalogo monografico Manuale tecnico 25

  31. Other products: 12 out of 25 (2001-2011) Market oriented Academically oriented “Products” for the “market” are marginal: other categories should be added

  32. CNR products in data base International journal literature Books Other means of transfer Enlightenment National literature

  33. Evaluation of SSHs: some comments Knowledgeisourconcern Do notforget the knowledge bundle and itsvariousfacets

  34. International Classification of Occupations (ISCO) 2111 0-12.10 Physicist2111 0-12.50 Physicist, acoustics2111 0-13.50 Physicist, astronomy2111 0-12.80 Physicist, atomic2111 0-12.20 Physicist, ballistics2111 0-12.60 Physicist, electricity and magnetism2111 0-12.70 Physicist, electronics2111 0-12.30 Physicist, heat2111 0-12.20 Physicist, hydrodynamics2111 0-12.40 Physicist, light2111 0-12.20 Physicist, mechanics2111 0-12.80 Physicist, molecular2111 0-12.80 Physicist, nuclear2111 0-12.50 Physicist, optics2111 0-12.20 Physicist, rheology2111 0-12.90 Physicist, solid-state2111 0-12.50 Physicist, sound2111 0-12.90 Physicist, theoretical2111 0-12.30 Physicist, thermodynamics 2113 0-11.10 Chemist2113 0-11.50 Chemist, analytical2113 0-11.40 Chemist, corrosion2113 0-11.40 Chemist, crystallography2113 0-11.20 Chemist, detergents2113 0-11.20 Chemist, dye2113 0-11.20 Chemist, food2113 0-11.30 Chemist, glass2113 0-11.30 Chemist, inorganic2113 0-11.20 Chemist, leather2113 0-11.30 Chemist, metallurgical2113 0-11.40 Chemist, nuclear2113 0-11.20 Chemist, organic2113 0-11.20 Chemist, paint2113 0-11.20 Chemist, petroleum2113 0-11.90 Chemist, pharmaceutical2113 0-11.40 Chemist, physical2113 0-11.20 Chemist, plastics2113 0-11.20 Chemist, polymer2113 0-11.50 Chemist, quality control2113 0-11.20 Chemist, rubber2113 0-11.20 Chemist, textile 22 19 3415 4-31.20 Representative, sales/technicalXXXXXXX Researcher: MISSING 1315 5-10.30 Restaurateur

  35. Evaluation of SSHs: some comments Knowledgeisourconcern Do notforget the knowledge bundle and itsvariousfacets Whyevaluate? Eveluationis a complex business, and bibliometricsas a shortcut • Much information on outputsof SSH isnotcoveredbybibliometrics • SSHs are structurallydifferentfromNSEs: differenttypesofevaluation Extending the SSH coverageofliteratureis OK, butthere are inherentlimits Significantdifferencesamongst CNR SSH DepartmentInstitutesdeservefurtheranalysis • More experimentationneedstobedone in SSH: in the meantimebecautiuous

  36. Research evaluation in SSH

  37. Thank you for attention

More Related