Knowledge Value Mapping Approach to Knowledge Translation: Its Relation to Evaluation and Management of R&D and Prac

1. Knowledge Value Mapping Approach to Knowledge Translation: Its Relation to Evaluation and Management of R&D and Practical Applications Juan D. Rogers School of Public Policy Georgia Tech KT-KVM Workshop, NCDDR

2. Presentation Outline • General Problems of R&D Evaluation • Use of Knowledge as Key Component in R&D Performance • Use-and-Transformation in Tech Transfer • Examples from Knowledge Translation Applications • Developing Knowledge Value Maps KT-KVM Workshop, NCDDR

3. General Problems of R&D Evaluation • Political climate demands a “mechanical control” model • Incompatible with well known traits of science: • More valuable results are the most unpredictable • Every so often changes its mode of operation to suit new challenges • We have poor understanding of the socio-cognitive mechanisms that cause many important things in the workings of research • Focus should broaden from “how” to “what” we are evaluating • Almost always happens in actual R&D evaluation projects • Normative analysis: what values are realized KT-KVM Workshop, NCDDR

4. Knowledge Use and R&D Policy Performance • Outcomes, impacts, ultimate results happen in a different location from knowledge creation • Social systems that must reflect impacts and outcomes are not the ones that receive resources to do research • Many dissimilar communities and social actors are involved: E. Rogers - heterophily • Complex knowledge flows are involved • Creates difficulties for R&D policy design and implementation and program management and evaluation • Accountability is hard to articulate: • Due to the displaced outcomes vis a vis research activities • Due to the poor understanding of the mediation processes  Effectiveness of transfer is inherent in R&D performance KT-KVM Workshop, NCDDR

5. Use-and-Transformation in Knowledge Dynamics • Knowledge Use and its cognates • Technology Transfer • Diffusion of Innovations • Knowledge transfer • Knowledge translation • Beyond the “Linear Model” • But how far? • Analytical frameworks reflect normative concern for impact rather than empirical knowledge of underlying phenomena • No pure use of knowledge in its flow • Knowledge is transformed as it takes each step in its flow • True of its process of creation • More serious and less understood as it crosses community boundaries • All relevant participants are hybrid user-producers of knowledge KT-KVM Workshop, NCDDR

6. Examples of Complexity of Knowledge Flows • Features of industrial networks affect change at the industry level (Andersson et al. 2006) • Knowledge flow from universities to industry: • Some indicator uses encourage counterproductive strategies (Langford et al. 2006). • Knowledge flow in internal networks for effective team work (Zhuge 2006). • Enthusiasm about the potential of collaborations • Collaborative work can be very difficult to carry out in practice (Dustdar 2002). KT-KVM Workshop, NCDDR

7. A Knowledge Value Mapping Approach: Aims • KVM focuses on the knowledge flows between knowledge creation activities and the impact contexts • Management of R&D always includes concern for knowledge flow • Internal to R&D process • Interdisciplinarity • Portfolio management • External to R&D process • Impact and outcomes • Mission requirements • Accountability to stakeholders • Knowledge flows are hard to manage • May be facilitated to certain extent • They are not uniform across relevant dimensions KT-KVM Workshop, NCDDR

8. A Knowledge Value Mapping Approach: Basis • Content-value structures affect knowledge flows • Knowledge doesn’t have all its consequences in itself • Current evaluation takes only goals and objectives as relevant values • When stakeholders are included, their values are taken as added or conditional objectives (i.e. explicit interests) • Process is substituted for value articulation • This perpetuates the fact-value dichotomy • Values must be investigated empirically in the entire domain of knowledge flow • Interests and agendas matter, but • Epistemic values such as validity and rigor criteria are generally not analyzed for their effect on outcomes (intra-content) • Values must be also pursued for different articulations KT-KVM Workshop, NCDDR

9. A Knowledge Value Mapping Approach: Procedure • Map the field • Identify creators and users (we call them “Knowledge-Value Collectives) • Explicit references (publications, citations, etc.) • Documented relationships (advisory roles, collaborations, consortia) • Nominations • Find the patterned dynamics and formal agreements (organized actions, teams, collaborations, contracts, etc.; we call them “Knowledge-Value Alliances”) • Find the communication patterns • Find evidence and infer the normative stances of all involved • Articulate relevant patterns of knowledge flow KT-KVM Workshop, NCDDR

10. A Knowledge Value Mapping Approach: Applications • Retrospective KVM: • Features of past or present knowledge flow • Examples: • Division of knowledge labor in interdisciplinary teams or inter-sector teams • Emergence of new specialized roles in interdisciplinary research • Institutional innovation in technology transfer • Community valuation of hidden knowledge outcomes • Prospective KVM: • Given KVM and proposed intervention, predict likely effects • Determination of facilitating and hindering factors for knowledge flows (relevant to evaluation and other policy concerns) KT-KVM Workshop, NCDDR

11. Application of KVM to Knowledge Translation in Health • Evidence of need for knowledge-value maps from the literature: • Diversity of uses and contexts • Insufficiency of research results for policy making • Diversity of types and mechanisms of use of research results • Variety of incentives in the context of use • Multidimensional matching of interventions and contexts in AIDS and Substance Abuse • Complexities in a single field application • Theoretical commitments in research • Costs of knowledge absorption • Professional identity of practitioners • Values at different levels of change • Contexts of practitioner decision making • Interdisciplinarity • Dissemination networks and effects on tech transfer KT-KVM Workshop, NCDDR

12. Underlying Value Issues in KT: Diversity of uses and contexts • Alignment with ideology facilitates adoption • KT begins with assessment of research and user’s preferences • “hire a knowledge broker to help find and develop those relationships” (Jacobson et al 2003) • Routine feedback of implementation lessons from the diversity of application contexts is key to effectiveness (Van Kerkhoff and Szlezak 2006) KT-KVM Workshop, NCDDR

13. Underlying Value Issues in KT: Insufficiency of research results • Complexity of the interface of research and policy making • Incentives for both communities are necessary for managing the interface (Lavis et al 2004; Lavis 2006) • Introduction of new guidelines is costly and benefits variable or ambiguous • Research results are not sufficient guidance for policy design and implementation • Extra knowledge is not within health researchers’ competency to produce (Grimshaw et al 2004) KT-KVM Workshop, NCDDR

14. Underlying Value Issues in KT: Diversity of types and mechanisms of use of research results • There are three categories of uses: instrumental, conceptual and symbolic. • There are many explanatory mechanisms on the features of knowledge that facilitate its use: • Engineering explanations • Organizational interest explanations • Two Communities explanations • Interaction explanations • The category of use and the mechanism varies across agencies and policy domains. (Amara et al 2004) KT-KVM Workshop, NCDDR

15. Underlying Value Issues in KT: Incentives in the context of use • Implementation of evidence-based practice (EBP) in the field of occupational therapy (Welch and Dawson 2006): • Practitioners in this field are embedded in a hierarchy • Most practice change is aimed at individual practitioners • Numerous obstacles for this process to succeed: • perceived labor unfairness, • misalignment of the EBP materials with their form of knowledge acquisition and application in the field, • lack of ownership of the content of EBP presented to them, • lack of structures of mutual support for engaging in learning new practice guidelines. • “…learning framework that focuses on the intended users of research as the most critical element of the process” (Farkas et al. 2003). KT-KVM Workshop, NCDDR

16. Underlying Value Issues in KT: Multidimensional matching in AIDS and Substance Abuse Cases • Organizational development methods to introduce EBP in substance abuse treatment (Amodeo et al 2006) • Addresses multiple dimensions of change in the application context • Does not address all dimensions of absorption • Substance abuse treatment programs (Bartholomew et al. 2007; Greener et al. 2007; Rowan-Szal et al. 2007) • Higher organizational functioning led to better client engagement • Also led to fewer barriers to training and greater staff satisfaction with training • Successful training programs were ones that received support from their organizations to do post-training trials • HIV prevention interventions (Eke et al 2006; Harshberger et al 2006) • Preparation during the research design to include contextual factors • Intervention implementation requires ongoing proactive technical assistance KT-KVM Workshop, NCDDR

17. Underlying Value Issues in KT: Complexities of a single field application • “Useful Field of View” technology for older drivers (Ball et al. 2003) • move it to the field and application in a community setting. • What was involved: • Establishing collaborations with health organizations in the area; • Adaptation of the protocol to the field setting • Training field personnel • Finding consenting participants for field trials • Multiple claims to ownership of the database • Answering further questions of cost-benefits for the health care system • Legislation changes for purposes of changing required tests at MVA KT-KVM Workshop, NCDDR

18. Underlying Value Issues in KT: Theoretical commitments in research • Theoretical shift in mental health research (“recovery vision”) has radical consequences for what counts as evidence (Anthony et al. 2003) • Fundamental treatment outcome variables used in the EBM approach become relatively unimportant • Others related to patients’ experiences and goals as indicators of recovery are more important • Requires a commitment to develop evidence and explore its implications • The very content of EBM will be different depending on the path researchers decide to follow in their theoretical commitment. KT-KVM Workshop, NCDDR

19. Underlying Value Issues In KT: More on epistemic values • Criteria that guide evidence assessment and quality change with field • EBM has its origin in two particular subfields of clinical medicine • General internal medicine and clinical epidemiology • Transferring evidence assessment criteria from them to other fields of medicine creates problems (Upshur et al. 2001). • EBM brings along the emphasis on quantitative measurement and, especially, of randomly controlled trials (Worrall 2002). • Outside of those fields these aspects of measurement and evidence gathering may not be feasible or as important: • “Although values are regarded as central to health-care decision making, the quantitative orientation to evidence proposed by advocates makes normative discourse seem an afterthought.” (Upshur et al. 2001) KT-KVM Workshop, NCDDR

20. Underlying Value Issues in KT: The cost of knowledge absorption • Getting involved with new knowledge requires a significant investment of capabilities and resources: • Concept of “absorption capacity” • Nature of efforts to enable the use of external knowledge • Firms have their own research capabilities with the objective of being able to absorb external knowledge (Cohen and Levinthal 1990). • Expectation of significant payoff for the commitment of resources to be made: • scientific research will be applied if it is perceived to add value to the user (Landry et al. 2006) KT-KVM Workshop, NCDDR

21. Underlying Value Issues in KT: Professional identity of practitioners • Values grounding the professional identity of nurses (Fagermoen 1997): • “Professional identity is defined as the values and beliefs held by the nurse that guide her/his thinking, actions and interactions with the patient.” • The actualization of other-oriented values is integral to their provision of nursing care • They realize self-oriented values, such as the satisfaction and reward in their work, through the realization of the other-oriented values. • Contextual factors limit their ability to provide care making their work less meaningful KT-KVM Workshop, NCDDR

22. Underlying Value Issues in KT: Values at different change levels • McGuire (1990) identifies 10 areas of difficulty for KT in nursing: • complexity of change process; • genesis of research programs; • formulation of research questions; • differences in theoretical approaches; • timescales and planning cycles; • information overload; • credibility; • applicability; • response to change; • management of change. • The target should not be at the individual level but at the organizational level: • Critique of focus on behavior modification KT-KVM Workshop, NCDDR

23. Underlying Value Issues in KT: More on levels of change • Rushmer et al (2004) show lessons for changes in practices in primary care come from the lessons learned in organizational change and organizational learning. • Many of the obstacles faced by KT are typical organizational change issues: • features of organizational leadership; • commitment to the program; • organizational culture; • openness and trust; and so on. KT-KVM Workshop, NCDDR

24. Underlying Value Issues of KT: Contexts of practitioner decision making • Decisions to adopt clinical practices by physicians are made in complex networks of consultation and collaboration with colleagues (Hader et al. 2007) • EBM guidelines expected to be followed independently • Must trust the judgment of unknown experts • Guidelines acquire meaning for the doctors within the entire network of relations and exchanges • Guidelines create questions about their ability to make critical medical decisions • Need reinforcement by trusted specialists • Consistent with potential patient compliance • Sustained support for the new practices • Not sufficient for doctors to produce new medical knowledge in the context of use. KT-KVM Workshop, NCDDR

25. Underlying Value Issues in KT: Interdisciplinarity • Empirical investigation in KT generally confined to the measurement of outcome variables related to the desired changes in behavior (Sudsawad 2007). • KT has been defined normatively first • Not the natural knowledge frontier of any particular field of research. • Numerous social science theories are relevant to KT (Graham et al. 2006; Estabrooks et al. 2006). • Education, diffusion of innovations, organizational behavior, psychology, social communication, management, political science, among others • Attempts at reduction to first principles or a small set of dimensions, • virtual new social science field of “KT research” KT-KVM Workshop, NCDDR

26. Underlying Values Issues in KT: Dissemination networks and effects on transfer of technology variations • Pringle et al. (2004) studied 5 health dissemination networks in US (RADAR; CADCA; CSN; NTCP; The Network) • Their ability to effectively disseminate federal-to-local information depends on fine tuning in four normative dimensions: Synergistic specialization; Brokerage; Coordination; Broad access information systems • These networks are different from those in other content areas and other institutional environments • Reisinger et al. (2007) evaluated three alignment systems based on two different alignment theories for fabrication of mono-limb prostheses • They had different performance outcomes, one theory faring better than the other • The different required equipment and processes led to different paths of technology transfer, i.e. different ease of transfer KT-KVM Workshop, NCDDR

27. Concluding Remarks • Knowledge flows are inherent in R&D evaluation • “Nobody’s land” between research and impacts • Must be determined empirically in the evaluation process • KVM is proposed to do so • Content-value maps of user-producers of knowledge KT-KVM Workshop, NCDDR