Bioethics research and the ERA: How can we make our research count in academia and in practice

1. Bioethics research and the ERA:How can we make our research count in academia and in practice Wendy Rogers, CAVE, MqUni Catriona Mackenzie, CAVE, MqUni Katrina Hutchison, CAVE, MqUni Ainsley Newson, VELIM, USyd

2. What is the ERA?: An exercise to identify and promote excellence in research in Australian higher education institutions (HEIs) To establish an evaluation framework providing assurance of the excellence of research in HEIs To provide a national stocktakeof discipline-level areas of strength and areas with opportunity for development To identify excellence across the full spectrum of research performance To identify emerging research areas and opportunities for development To allow for national and international comparisons of Australia’s research for all disciplines Excellence in Research for Australia 2012, National Report, ARC

3. What is the ERA used for? • ERA is a key performance indicator in the 2011-13 mission-based compact between the Australian gov’t and the HEIs. The ERA has or will inform: • the Research Workforce Strategy • the 2014-16 mission-based compacts • the allocation of some funding through the Sustainable Research Excellence in Universities initiative • the new minimum standards for higher ed research and training administered by TEQSA

4. Commonwealth research objectives “To progressively increase the number of research groups performing at world class levels as measured by international performance benchmarks” “To promote collaboration, including collaboration between researchers within Australia and internationally” MqUni mission compact 2011-13, p 26

5. MqUni mission compact “We will consider the ERA results in the context of our research and research training strategies to determine where additional resources should be made available to ensure we achieve world standing in all of our areas of research” • MqUni mission compact 2011-13, p 28

6. 2012 ERA indicators Indicators of research quality: publishing profile, citations, ERA peer review, peer-reviewed research income Indicators of research volume and activity: total research outputs, research income, other research items Indicators of research application: research commercialisation income and other applied measures Indicators of recognition: based on range of esteem measures

7. Discipline specific indicators: quality and volume # Field of Research 11: Medical and health sciences 1801: Law 2201: Applied ethics 2203: Philosophy * Books, book chapters, journal articles, conference publications, non-traditional research outputs

8. Discipline specific indicators: esteem 11: Medical and health sciences 1801: Law 2201: Applied ethics 2203: Philosophy

9. Discipline specific indicators: applied measures 11: Medical and health sciences 1801: Law 2201: Applied ethics 2203: Philosophy

10. 2012 ERA results: FoR 22 Philosophy and Religious Studies

11. FoR 11 and 18 contribution to the national landscape Medical and health sciences Law

12. 2012 ERA results: FoR 2201 Applied ethics

13. 2012 ERA results: FoR 18 Law

14. ERA-related challenges for bioethics research • Neither fish nor flesh nor good red herring* • Not accepted/counted for home department FoR • Not considered scholarly/of high quality by discipline specific peer review/panel • Tensions between: • Performing relevant and collaborative research • Communicating with appropriate audiences • Turning inward to FOR-specific journals • Mismatch between esteem measures for philosophy and law and likely achievements of bioethicists • Low overall rating of Applied Ethics cf Law and Philosophy *The Proverbs of John Heywood, 1546

15. Case Study: ARC Linkage Project On the Cutting Edge - Promoting Best Practice in Surgical Innovation(ARC LP110200217)

16. Project Team by discipline(at Macquarie University unless specified) Chief Investigators • Prof Wendy Rogers (bioethics: philosophy and ASAM) • Dr Jane Johnson (philosophy) • DrMiannaLotz (philosophy) • Prof Denise Meyerson (law) • Mr George Tomossy (law) • Prof Guy Maddern (surgery, University of Adelaide) • Prof Tony Eyers (surgery, ASAM) Partner Investigators • Dr Angela Ballantyne (bioethics, University of Otago) • Prof Colin Thomson (Law, UoW and Houston Thomson) Postdoc + RA • Dr Katrina Hutchison (philosophy) • MsSwantjeLorrimer-Mohr (philosophy)

17. Weaving a research thread: multiple strands and multiple audiences for the outputs of each strand Law and regulation of surgical innovation Ethics of surgical innovation Partner Relations Taxonomy: what is innovation? Qualitative research: views of stakeholders Admin and logistics

18. Publications 2012/13 Innovative surgery: the ethical challenges J Johnson & W Rogers, Journal of Medical Ethics 2012; 38: 9-12. Controversy Over Vertebroplasty J Johnson, W Rogers & L Jeffree, Journal of Medical Imaging and Radiation Oncology 2012; 56: 449–451 Innovative Surgery and the Precautionary Principle D Meyerson, Journal of Medicine and Philosophy (accepted 8/3/12) Addressing within-role conflicts of interest in surgery W Rogers and J Johnson, Journal of Bioethical Inquiry 2013; 10 (2): 219-225. Identifying surgical innovation: A qualitative study of surgeons’ views Rogers WA, Lotz M, Hutchison K, Pourmoslemi A, Eyers A. Annals of Surgery(in press) What Chappel v Hart really stands for and some ramifications for innovative surgery. D Meyerson. ANZ Journal of Surgery(in press) Reconceiving early-stage surgical innovation as sui generis surgical research M LotzTheoretical Medicine and Bioethics(accepted)

19. Other related papers The ethical challenges of innovative surgery - a response to the IDEAL recommendations. J Johnson, W Rogers, M Lotz, C Townley, D Meyerson, G Tomossy. Lancet 2010, 376 (9746): 1113-1115. What Can Feminist Epistemology do for Surgery? M Walker, W Rogers. Hypatia (accepted). Defining surgical innovation: a "family resemblance" concept? K Hutchison, W Rogers, A Eyers, M Lotz. Submitted to Journal of the American College of Surgeons(under review).

20. Journals and FoRs in which Linkage outputs are published

21. Journal ranking of Linkage outputs by IF and journal rankings

22. Journal ranking of Linkage outputs by ERA 2010 list

23. The RAE/REF in the United Kingdom What has been the experience for academics in bioethics? Centre for Values, Ethics and the Law in Medicine | The University of Sydney Dr Ainsley Newson | Senior Lecturer in Bioethics

24. The UK ‘Research Excellence Framework’ (REF) • Assesses quality of research in HEIs in UK via expert review • Professional judgement of discipline-based expert panels (36 Units of Assessment (UoA) divided into 4 main Panels), each referencing a common framework • 2008-2013; census date 31 Oct 2013. Results 2014. • Three purposes to the REF: • Assessment outcome will inform selective allocation of research funding to HEIs from 2015-16; • Provide accountability for public investment in research and produces evidence of the benefits of this investment; and • Provide benchmarking information and establish reputational yardsticks

25. The UK ‘Research Excellence Framework’ (REF) • Three elements to a REF submission (result per panel, per institution): • Outputs (65%): assess the “originality, rigour and significance” of submitted research outputs, with reference to international quality standards. Four submitted outputs per staff member. • Some panels use citation data in quality assessment • Journals not ranked, but individual outputs graded on a 5-point scale of quality • Impact (20%): the “reach and significance” of impacts on economy, society and/or culture that were underpinned by excellent research undertaken by the UoA since 1993 • Environment quality (15%): its “vitality and sustainability”; includes equality and staff development

26. How is bioethics assessed in the REF? • Several places where bioethics could go: • Panel A, UoA 2: Public Health, Health Services and Primary Care* • Panel C, UoA 20: Law • Panel C, UoA 22: Social work and social policy • Panel C, UoA 23: Sociology • Panel D, UoA 32: Philosophy (which has at least some members who may be ‘sympathetic’ to bioethics research) • Panel D, UoA 33: Theology and Religious Studies * = this sub-panel will make use of citation data • Problem: bioethics research does not have its own sub-panel • Problem: bioethics research gets rejected (internally) for most panels

27. Has anyone pointed this issue out? • Kong et al (Institute of Medical Ethics): BMJ Letter (2011;342:d3968 ) • “Conceptual error” to exclude theoretical medical ethics • Given central part of ethics education in medical schools, poor that it is not reflected in valuing research excellence • Preventing integration of sciences and humanities • Result: no change / no response • Concern lack of REF recognition will deter researchers from entering field

28. Example • Can affect job prospects: • Recent Senior Lecturer/Lecturer in Global Ethics job at the University of Birmingham, UK: • “Candidates should be able to demonstrate achievement against the following criteria: • a strong research record including evidence of independent research in high quality (strong REF) publications in philosophy and support and recognition as appropriate to level is already clearly evident. • (Please note that while global ethics is multidisciplinary the successful candidate will need to be eligible for inclusion in the Philosophy REF and is likely to have expertise in applied philosophy, moral philosophy and/or political philosophy.)”

29. ‘Straw poll’ of UK colleagues’ thoughts on REF • Many academics in bioethics experience problems in being returned • Medical Schools are rejecting bioethics research that is not empirical • Philosophy departments are rejecting bioethics research that is not advancing theory in philosophy • Some (senior) academics are attempting to get submitted to different panels • Some (senior) academics are attempting to get submitted under Impact alone

30. Twitter discussion, early June 2013 Some UK Colleagues’ views: • “applied ethics is rumouredto get a better hearing in philosophy since Heather [Widdows] is on the panel - that said everyone seems to be doing their own thing and still avoiding that panel - I heard Lancaster… is going in Social Policy instead…” – David Hunter • “some bioethics goes into law. But - as usual - it will likely suffer _wherever_ it goes” – Richard Ashcroft • “and some in education, but share Richard’s concerns about subject's consideration” – Deborah Bowman • “wherever we're put, the dominant view is "all very interesting, but not Philosophy"... Or law, social policy, etc. The publications are too short, too long, wrong method.Too many authors, too few, too discursive, not enough data etc. etc. etc. So in sum: I hate the whole charade.” – Richard Ashcroft

31. Personal experience Participated in RAE (2008) and internal mock REF (2012) • Was returned under 2008 RAE, albeit with 2 publications due to ‘early career’ status, despite having 4 eligible publications • 2012: • Not submittablefor the Health Services Research Panel. Informed that the publications lacked sufficient empirical work • Not submittable to the Philosophy Panel. Informed that the publications do not advance philosophical theory • Rather they apply it to examples in medicine and health care • Bristol academic is chairing the Philosophy UoA… • Affects morale – i.e. working in a discipline that is not recognised, nor understood by those outside of it?

32. The difficulties of ‘Impact’ • Definition of Impact (20%): “An effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” • Includes (but not limited to) effect, change or benefit to: • The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding • Of an audience, beneficiary, community, constituency, organisationor individuals • In any geographic location whether locally, regionally, nationally or internationally • Impact has to: • Occur during the relevant time period (5-year window) • Be underpinned by your own published research (20 year time window) • Underpinning research needs to be 2* or higher in quality (i.e. recognised internationally, internationally excellent or world leading) • Demonstrated via statement and submission of case studies

33. REF: ‘Impact’ How we were told to try to impact: • NICE guideline citations – demonstrate change as a result of research • Evidence of changes in clinical or policy practice, especially where there can be quantified • Check the NHS Map of Medicine (http://eng.mapofmedicine.com/evidence/map/index.html) to see if your research is referenced. • Department of Health reports, policy statements and even minutes of meetings which cite your research. • Check guidelines of other counties (e.g. US, Netherlands). • Press and media publications

34. Can bioethics make an ‘Impact’?? Case study: • X is renowned in bioethics for her work on A. This leads her to being appointed to a relevant commission examining policy/law on A. That commission recommends a particular policy response, which is taken up by the government. • How can X demonstrate this impact in the REF? • X’s research needs to be linked explicitly to the policy/law in question • X’s research needs to be 2* quality or above (internationally recognised) • A lot of the ‘engagement’ work we do in bioethics may be too general to meet these more measurable/quantifiable/tangible standards Thanks to Dr Richard Huxtable, University of Bristol

35. Lingering questions… • How should ‘quality’ in research be defined? • Is one approach to fit all disciplines appropriate? • Should impact only be that which is quantifiable? • What should be done about institutional game playing? • ‘On paper’ bioethics has plenty of places to ‘count’, but institutions don’t follow this locally • What should count as a citation? And who should measure it? Be mindful of biases in citation • E.g http://kieranhealy.org/blog/archives/2013/06/19/lewis-and-the-women/

36. Measuring impact (Australian Government discussion paper June 2013) • Impact understood as ‘benefit’: ‘positive economic, social and environmental changes that can be attributed to university research’ • Engagement: ‘the pathways from university research activities to uptake and adoption of research outputs by research users and the realisation of subsequent economic, social and environmental benefits. These pathways may encompass activities (such as knowledge transfer and dissemination), policy frameworks, governance arrangements and skill development’

37. Measuring impact. Aims? ‘Demonstrate the public benefits attributable to university research Identify the successful pathways to benefit Support the development of a culture and practices within universities that encourage and value research collaboration and engagement Further develop the evidence base upon which to facilitate future engagement between the research sector and research users, as well as future policy and strategy.’

38. Measuring impact: 1) Metrics Category 2, 3, 4 research income Patents Licenses # staff in research commercialisation # staff & postgrads in start up companies Engagement: online publications, professional & applied research publications $$s and numbers from research engagement events

39. Measuring impact: 2) Case studies • Minimum of 5 per institution, set maximum • Reported using 4 digit FoR codes plus SEO (socio-economic objective) codes • Assessed by panels consisting of at least 70% research users • Assessment in terms of reach (spread or depth of benefit); significance (intensity); contribution (of research to reported benefit); validation (corroboration of impact claims) • Selected case studies reported in dedicated report, MyUniversity website, mission-based compacts.

40. Case study on impact: Submission for Linkage to NHMRC 2011 consultation on “Identifying and Managing Conflicts of Interest: Draft Policies …” Five points from our submission are reflected in the 2012 final version of these guidelines: Include within role conflicts: 1.4.2(f) new section on within role interests Try to find experts without conflicts: 1.2.2 acknowledges that technical knowledge should not override all other considerations; and 1.2.4: make efforts to find experts without conflicts Provide education on CoI: 3.1.2 mentions education of appointees on CoI Criticism of $$ thresholds: thresholds now omitted Appoint chair with no (cf“no major”) conflict: 1.5.1(d) chair to have no CoI ….

41. What can we do about it? • Engage: • Know your FoRs, journal codes and the rules for reassigning these (Exemptions under 5.4.3.1) • Be strategic (double publish) • Plan strategy with department head and ERA leaders • Catalogue potential measures of impact • Committees, meetings • Commissioned reports • Submissions, eg to legislation reviews • Track any influence/mention

42. Thank you