The RAE is Dead! Long live the REF!
The Changes. • The RAE looks at three main areas: • Outputs • Environment • Esteem • We are used evaluations of Environment and Esteem being “informed” by metrics • The intention is that under the REF all three areas will be metric determined. • This will require the introduction of bibliometrics into the REF.
The Three Most Important Words • You might guess that the three most important words in the REF were: Research Evaluation Framework Unfortunately, you would be wrong!
The Three Most Important Words • Without any doubt the three most important words in the REF are: We Don’t Know! The worrying thing here is that, at the moment, “WE” includes HEFCE!
Surely HEFCE aren’t that stupid? • HEFCE have conducted a pilot study into gathering bibliometric data. • The pilot was to run concurrently with the 2008RAE to provide a basis for comparison. • Unfortunately this proved to be impossible – and so no benchmark is available!
The HEFCE Bibliometric Pilot Study • Involved 16 universities • Covered 35 UoA’s – good coverage in; Science, Computing, Engineering. Very little social science or arts coverage. • The candidates were asked to provide bibliometrics – including citation data on all staff in the UoA which were submitted to RAE2008. • The candidates were asked to provide the information as quickly as possible, consistent with quality checking.
The HEFCE Bibliometric Pilot Study • The “best” university managed to comply in 3 days! • One university took 150 days! – using an undisclosed, but substantial, number of staff. • In two years time we could be asked to do this on an annual basis!
Lessons Learnt from the Pilot. • Difficulties with older or existing information. • The process must be on-going and not just pre-submission. • Data cleansing is very time consuming – formats, mistakes, missing fields etc. • Difficulties linking publications to staff and staff to UoA’s (HESA id). • In some subjects the coverage simply is not there – i.e. the information does not exist!
Does this slow HEFCE down? • Not a bit of it! • Autumn 2009 – consult on main features of REF • May 2010 – outcome of consultation, phasing in timetable, main operational features in place. • 2011-12 – metrics begin to inform funding. • 2012 – submissions to 2013 ‘light touch’ peer review process. • 2013 – undertake full assessment using ‘light touch’ peer review and metrics • 2014 – HEFCE research funding for all subjects driven by REF from this time on.
So, what should we do? • JMU Working Party set up. This includes; • UoA co-ordinators from science/technical, social science and arts subject areas. • Colleagues from Computer Services and the Library. • Representatives from Research Office. • Decided that we must plan for the “worst case” scenario. That is, we will be required: • to return bibliometric data on ALL publications, • by ALL academic staff, • referenced back to UoA’s, • Over a 10 year period, • annually!
Introducing...Sympletic • Sympletic is a publications management system developed in collaboration with Imperial College. • What Sympletic does: • We supply a list of names, aliases, email addresses, affiliations and UoA data. • Sympletic trawls databases looking for publications by these individuals. • If it finds a candidate publication it emails the person concerned and asks them to confirm if the paper is theirs.
Introducing...Sympletic • On confirmation... • Sympletic will download all information regarding that publication. • Sympletic will maintain this information in an updated condition – particularly citation data. • Staff will be able to examine their own record and notify Simpletic of any errors or omissions.
Sympletic • The plus points: • Once set up, the system is self maintaining. • Producing up to date reports at, multiple levels, • University • Faculty • School • Research Group • Individual • Can automatically update websites • Resource for REF, CV’s, Grant Applications, Corporate Publications etc. • Sympletic “talks” to Oracle and Oracle HR.
Sympletic • The minus points: • We need a definite staff list – referenced to UoA’s • Staff engagement • Personal benefits • Managerial benefits • Issues with coverage – WoS and PubMed. Other databases? • Does not solve the problem of the data simply not existing.
Sympletic – Action Points • Software Demonstration • If used the timetable would be: • Spring 2009 – Pilot • Sept 2009 – roll out • March 2010 – full deployment. • Management input needed to promote staff engagement by adopting Sympletic for other processes: • Profs/Readers • Website maintenance • Validation/Accreditation documents • PDPR