Realizing the Roberts Agenda: KPIs

1. Realizing the Roberts Agenda: KPIs Chris Park Director of the Graduate School, Lancaster University Senior Associate, Higher Education Academy Chair, the Rugby Team Co-ordinator, QAA Special Review of RDPs

2. The skills agenda • What? • generic and research skills • Who? • early career researchers • Research students and post-docs • Why? • Personal and professional development • Employability • Knowledge transfer/economy UK GRAD Yorks & NE Hub - Dealing with KPIs

3. Drivers of change • National requirements • Research Councils Joint Skills Statement • Revised QAA Code of Practice • Roberts Review • Additional funding • Roberts money • Internationalisation • Bologna, European HE Area • Lisbon, European Research Area UK GRAD Yorks & NE Hub - Dealing with KPIs

4. Who cares? • National • RCUK, Treasury • Regional • UK GRAD hubs, collaboration • Local • Approach, resourcing and delivery by HEI • Individual • Benefits to the individual research student or post-doc UK GRAD Yorks & NE Hub - Dealing with KPIs

5. Roberts 101 • SET for success [2002] • “… the minimum standards should include the provision of at least two weeks’ dedicated training a year, principally in transferable skills” • Career development for PGR students and postdocs • Funding • Around £20 million a year at ‘steady state’ • HEI allocation proportional to number of RC-funded research students and post-docs • Accountability • Annual Roberts Reports from HEIs to RCUK UK GRAD Yorks & NE Hub - Dealing with KPIs

6. Rugby Team 101 • Establishment • Jan 2005 UK GRAD Roberts Policy Forum • Sector representation • Individual HEIs, HEFCE, RCUK, QAA, NPC, UKCGE, UK GRAD, UUK, HEA, UK HERD • Mission • “to propose a meaningful and workable way of evaluating research skills development in researchers” • Mode of operation • Core team [meetings, email discussion] • Virtual consultative group [email, UK GRAD bulletin board] UK GRAD Yorks & NE Hub - Dealing with KPIs

7. Evaluation - rationale • Feedback to funding bodies such as HEFCE, RCUK • Inform enhancement of quality of experience for researchers by HEIs • Assess impact of recent initiatives on employability of PGRs and post-docs • Monitor engagement of HEIs and the sector in skills development agenda UK GRAD Yorks & NE Hub - Dealing with KPIs

8. Evaluation - approaches • Managerial • Evidence-based decision-making • Performance management – KPIs • Focus – external, compliance and accountability • ‘Softer’ approach • Evidence-based decision-making • Understanding of processes and impacts • Focus – internal, student experience and enhancement UK GRAD Yorks & NE Hub - Dealing with KPIs

9. KPIs favoured by HEIs • First year Roberts Reports • Content analysis of 77 HEIs • Participation in research training programmes • Participant evaluation of training activities • Progression and completion rates • Improvement in employability UK GRAD Yorks & NE Hub - Dealing with KPIs

10. KPIs – some possibilities • INPUT • All courses on institution-wide accessible database • All courses mapped against JSS • THROUGHPUT • % of PGRs who undertake TNA • % of post-docs who undertake TNA • % of PGRs who engage in PDP process • % of post-docs who engage in PDP process UK GRAD Yorks & NE Hub - Dealing with KPIs

11. OUTPUT • Confidential feedback mechanisms for – • PGRs • Post-docs • Supervisors • Leavers • alumni • Survey of employers’ views UK GRAD Yorks & NE Hub - Dealing with KPIs

12. OUTCOME • Institutional submission rate • Institutional completion rate • Institutional non-completion rate • No of refereed papers per PGR • No of refereed papers per post-doc • First destination information UK GRAD Yorks & NE Hub - Dealing with KPIs

13. Roberts Reports, 2006 • Summary of spend 2005-6 • Examples of effective practice on UK GRAD database of practice • Summary report • Management of RTP • Career development and training opportunities • Development of research training and skills development • Sharing processes and practices UK GRAD Yorks & NE Hub - Dealing with KPIs

14. Other sources of evidence • QAA Special Review • Overview Report • Good practice and areas for further consideration, by precept • Includes skills development and research training • Dissemination • Key lessons - Jan 07 Roberts Policy Forum • Present report – Feb 07 UKCGE Winter Conference UK GRAD Yorks & NE Hub - Dealing with KPIs

15. HEA PRES • Postgraduate Research Experience Survey • Online survey (BOS/CROS) • Core set of questions; HEIs add own • Informed by QAA CoP and RCUK skills requirements • Summer 2006 pilot – 8 HEIs • No league tables • HEI sees own results and aggregate • Benchmarking • National launch March 2007 UK GRAD Yorks & NE Hub - Dealing with KPIs

16. HEI approaches to evaluation • RT06 review of institutional practices and approaches • Resource to inform sharing of effective practice UK GRAD Yorks & NE Hub - Dealing with KPIs

17. Roberts – the future? • Not yet clear • Comprehensive Spending Review • Likely outcome • Continued funding on sustained basis? • Mode of allocation to HEIs unchanged? • Not necessarily ring-fenced [through HEFCE grant?] • Future reporting requirements? UK GRAD Yorks & NE Hub - Dealing with KPIs

18. What next? • Rugby Team recommendations • “That RCUK should … • “fund a national survey of the extent to which researchers and academics have engaged with the skills development agenda, and run it during 2007” • “agree a stable format for future HEI reports on their use of Roberts monies, and make this known to the sector by late summer 2007” • “scope and fund a project designed to answer the question ‘what is the impact on the UK economy of the skills development funded by Roberts money?’, beginning during 2007, to be carried out on a consultancy basis by someone with appropriate expertise.” UK GRAD Yorks & NE Hub - Dealing with KPIs