Praxis or Poiesis ? On the Assessment of Applied and Practice-Based Research

1. Researching PraxisPedagogy Culture and Society seminarGothenburg University Saturday 13th September Praxis or Poiesis? On the Assessment of Applied and Practice-Based Research Dr Alis Oancea University of Oxford

2. What is applied research? • How may applied and practice-based research be assessed? • How can we begin to describe ‘good’ applied research and its relationship to practice and policy?  three domains of excellence • Are the three domains compatible? Is it justifiable to expect that they can all offer legitimate guidance to assessment?

3. 1. What are applied and practice-based research?

4. Demonstrable knowledge (37) SCIENTIFICALLY BASED RESEARCH- The term scientifically based research' — (A) means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs; and (B) includes research that — (i) employs systematic, empirical methods that draw on observation or experiment; (ii) involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn; (iii) relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators; (iv) is evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random- assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls; (v) ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and (vi) has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review. • No Child Left Behind – 8 Jan 2002 (US)

6. Technical knowledge OECD 2002a: 78 ‘original investigation undertaken in order to acquire new knowledge.., directed towards a specific practical aim or objective’ Stokes, 1997

7. an area situated between academia-led theoretical inquiry and research-informed practice, and consisting of a multitude of models of research explicitly conducted in, with, and/or for practice.

8. “not to disregard reason and principle, and yet to attend to them only within the context of an even closer attention to concrete situations” (Dunne, 1993)

9. “therefore we ought to attend to the undemonstrated sayings and opinions of experienced and older people or of people of practical wisdom no less than to demonstrations; for because experience had given them an eye they see aright” (Aristotle, EN, 1143b, 10-15)

10. 2. How is applied and practice-based research assessed?

11. Current climate: the RAE ‘Where researchers in higher education have undertaken applied and practice-based research that they consider to have achieved due standards of excellence, they should be able to submit it to the RAE in the expectation that it will be assessed fairly, against appropriate criteria’ UK Funding Bodies (2004, para 47) • RAE subject panels to ‘define appropriate criteria for identifying excellence in different forms of research endeavour, while attaching no greater weight to one form over another; and (…) to make provision to recognise the diversity of evidence for excellent research’ (UK Funding Bodies 2005, para 16).

12. Current climate: the REF • The RAE, despite ostensibly supporting applied research, may have contributed to the reinforcement, rather than the solution, of these problems (see criticisms in the Roberts and the Lambert reports of 2003) • Is the new REF likely to solve this problem? • Scale and funding • Citation patterns

13. The 2005 project: not another set of criteria, but • (a) recognize the diversity of perspectives on applied and practice-based research quality in education, and of the ways in which they build their legitimacy; • (b) preserve the importance of methodological and theoretical soundness in applied, as well as in ‘curiosity-driven’, research; • (c) emphasize the principle that education research ought to be assessed in the light of what it wants and claims to be, and not through a rigid set of universal ‘standards’.

14. Is research evaluation really a matter of getting the technologies “right” (i.e. of making them effective, controllable, and in perfect fit with their object)? • What are the dangers entailed by the focus on the fine-tuning of techniques and criteria to objects (and the reverse), rather than on the limits of the discourse from which these techniques emerge? • Can we help unsettle current discourses and shift the overall way of conceiving of research assessment and of research quality from a narrowly defined technical concern towards a wider (historically and philosophically) understanding?

15. Discursive alternatives to instrumental accounts of research assessment Alternative discourses  Complex entanglement of research and practice and different modes of knowledge Nurturing excellence/ virtue (epistemic, technical, and phronetic) Deliberation and judgment Assessment techniques help research communities to gain increased control over the contingencies of their practice • Current discourse • Hierarchical relationship between modes of research • Quality assurance and quality assessment • Quantification, measurement and ranking of performance • Assessment techniques unquestioningly produce externally-specified outputs

16. 3. How can we begin to describe ‘good’ applied research and its relationship to practice and policy?

17. domains of knowledge (or of engagement with the world: theoresis (contemplation); poiesis (production); praxis (virtuous action in the public space) • excellence or ‘virtue’: • episteme theoretike (knowledge that is demonstrable through valid reasoning); • techne (technical skill, or a trained ability for rational production); • phronesis (practical wisdom, or the capacity or predisposition to act truthfully and with reason in matters of deliberation, thus with a strong ethical component) (Aristotle, EN, 1139a, 27–28, 1178b, 20–22).

18. Episteme—demonstrable knowledge • Application as gradual gliding from the general to the particular, and from the abstract to the concrete. • Not the exclusive domain of “basic” research • Methodological concerns

19. Techne - technical skill • using research-based knowledge to attain specific, externally defined, ends or to bring about certain states of affairs, by controlling the resources and procedures involved; not necessarily instrumental • Control, systematicity, precision, explanation: “Universality and explanation yield control over the future in virtue of their orderly grasp of the past; teaching enables past work to yield future progress; precision yields consistent accuracy, the minimization of failure” (Nussbaum, 1986, pp. 97)

20. Phronesis - practical wisdom First-person action on internal principles of excellence: “conduct in a public space with others in which a person, without ulterior purpose and with a view to no object detachable from himself, acts in such a way as to realise excellences that he has come to appreciate in his community as constitutive of a worthwhile way of life” (Dunne, 1993, p. 10) Ethical deliberation about ends and reflective choice: “in art, he who errs willingly is preferable, but in practical wisdom, as in the virtues, he is the reverse” (Aristotle, EN, 1140b, 20-25)

21. Paradigm-dependant criteria Transparency/ explicitness Propriety Trustworthiness Advancement of knowledge Plausibility Engagement Reflexivity, deliberation and criticism Receptiveness Transformation, personal growth Quality domains Domains Marketability and competitiveness Cost Auditability Feasibility Added value/ ‘brand’ Fitness to purpose Salience/ timeliness Specificity and accessibility Concern for enabling impact Flexibility and operationalisability

22. 4. Are the three domains compatible? Is it justifiable to expect that they can all offer legitimate guidance to assessment?

23. Generalisable knowledge and teachability • Ethics and educational transformation • Experience and professionalism ?

24. Questions • What concept of the relationship research-practice/policy underpins your project? • Which of the domains of the framework (or others) does your work foreground? What does this entail in relation to the understanding of "good" research built into your work? • Which of the domains of the framework would least match your research?