Evaluation of Informatics Tools in Primary Care

1. Evaluation of Informatics Tools in Primary Care Frank Sullivan, Prof. of R&D TCGP, Dundee Liz Mitchell, Research Fellow, Glasgow Claudia Pagliari, Lecturer in Psychology, Dundee f.m.sullivan@dundee.ac.uk h.c.pagliari@dundee.ac.uk edm1a@clinmed.gla.ac.uk

2. Informatics The study of the acquisition, processing and use of information. Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer 1997. Informatik Informatique

3. Primary Care in the Information Age Moving from • Popper world 2 • Notions, intuitions, judgements, mystique • Popper world 3 • Objective reality open to criticism and logical correction LL Weed. Clinical judgement revisited Meth.Inf. Med 1999;38:279-86

4. GP Computer Screens and Prompts

5. Reference Information including guidelines Information Age Consultations Education Recall Retrieval Electronic Medical Record Patient Clinician Consultation Sullivan FM, et al

6. Problem 1: Limited evaluation of informatics tools • Failure to evaluate new resources is a major problem • Often top-down, technologist/manager - driven development, with little involvement of end-users in the process. • Expensively developed tools often discarded due to unanticipated technical difficulties or ‘people and organizational issues’.

7. Problem 2: Inappropriate evaluation RCTs aimed at measuring ‘hard’ clinical and economic outcomes may not always be appropriate for informatics systems because a) they are not drugs but multifaceted procedural interventions and b) the type of questions asked of an informatics evaluation are broader, dealing as much with end-users’ acceptance and use of the system as with external outcomes.

8. Thinking about the WHOLE • The RCT may provide useful information but it can only give part of the story • Comprehensive evaluation of health informatics tools requires a broader range of research methods, involving both quantitative and qualitative approaches. • The ideal method, or combination of methods, will be determined by the research questions and the context and timeframe in which it is taking place.

9. Which research questions/whose perspective? Does it work? Will they use it? Developer Purchaser User Is it fast? Is it fun? What is the cost:benefit? Patient Is it safe? Will it work?

10. 2 Classes of research method for informatics evaluations Objectivist: concerned with objective assessment of clearly defined variables, usually measured quantitatively (e.g. via experimental or correlational studies). Subjectivist: based on the judgements of expert evaluators, system users, potential users or other stakeholders. Often rely on qualitative, anthropological research methods. Friedman and Wyatt, 1997

11. Objectivist approaches 1 Comparison-Based: Employs experiments and quasi experiments. Comparisons based on small numbers of outcome variables e.g. Hypothesis: “Compliance with guideline recommendations to check diabetics’ feet annually will increase following introduction of computer-based reminders system”

12. Objectivist approaches 2 Objectives-Based: Aim is to determine whether the resource meets its designer’s objectives. E.g.: Are fully integrated patient records accessible to the GP within 2 minutes?

13. Objectivist approaches 3 Decision Facilitation: Focus on answering questions important to developers and administrators. Usually used in formative studies when developing new resources. e.g. Systematic study of various formats for a presenting guideline information on-screen, conducted as part of the process of resource development.

14. Objectivist approaches 4 Goal-Free: Evaluators are blinded to the intended effects of the resource and must chart all its effects. Aims to reduce reporting bias and to uncover both unintended and intended affects. E.g. Conducting patient chart reviews before and after introduction of an information resource without telling the reviewer anything about the nature of the information resource.

15. Subjectivist approaches 1 Responsive-illuminative: Focuses on the reports of users, e.g. feedback following a demonstration or period of hands-on familiarisation with the tool. Useful for technical troubleshooting and for examining contextual factors which may affect implementation. E.g. Observations of prototypical users in a laboratory setting, followed by one-to-one interviews about the advantages and disadvantages of the resource and discussion of what has been observed.

16. Subjectivist approaches 2 Art Criticism: Analysis and review of a resource by a generic expert. E.g. Software review in a technical magazine. Inviting a noted consultant on user interface design to spend a day on site to offer suggestions regarding the prototype of a new system.

17. Subjectivist approaches 3 Professional review: Management consultancy type approach using extended site visits by experienced peers to the environment in which the resource is installed. May employ a combination of methods including speaking to users, observing the system in operation etc. E.g. A site visit by a government review team to several research groups competing to have their patient management screens for asthma adopted nationally.

18. Subjectivist approaches 4 Quasi-legal: Mock trial or other formal adversarial procedure to judge a resource. Rarely used. E.g. Staging a mock debate at a research group retreat.

19. Tailoring methods to the problem Comprehensive evaluation may require a combination of research methods involving both objectivist and subjectivist approaches. The choice will relate to the specific research questions and the stage of the evaluation.

20. General steps in informatics evaluations • Define and prioritise study questions • Define the "system" to be studied • Select or develop reliable, valid measurement methods • Design the demonstration study • Choose the appropriate methodology • Ensure that study findings can be generalized • Carry out the evaluation study (NB. Demonstration and evaluation phases may overlap)

21. Step 1: Define and prioritise your study questions • Decide exactly what you want to find out and specify your objectives. • Ideally questions should be be agreed between the research team, system developers, clinical & non-clinical users & patients. • Find out what has been done before

22. Step 2: Define the "system" to be studied • Is the system simple or multifaceted? Is it one component or the system as a whole that is of interest? If the former, can you isolate and evaluate that part alone? (e.g diabetes web-suite) • Develop a model for the evaluation to test. Results can be compared with the model to define the place of the new technology and further refine the model.

23. Step 3: Select or develop reliable, valid measurement methods • The aim of so-called ‘measurement studies’ is to ensure that the tools you use to assess outcomes are of as high quality as the methodology allows. • Try and use established measurement tools (e.g. questionnaires) if available. If not, there are clear procedures for developing them (see Friedman & Wyatt p71). • It may necessary to consult widely, interview potential system users individually or in groups in order to determine which are the key variables to be studied.

24. Step 4: Design the demonstration study • Leaving the evaluation until after a system is in place restricts the degree to which the results of the evaluation can be used to modify the system, resulting in less-than-ideal implementation (meaning not only access but also acceptance and use). • Gold standard approach to evaluation involves a ‘prototyping’ phase or ‘demonstration study’, in as realistic a context as possible, followed by one or more user-informed iterations of the system (i.e. ‘the evaluation-development cycle’). • May assess several objective and subjective variables including usability; attitudes; ideas for change; barriers to implementation.

25. Step 5: Choose the appropriatemethodology • Approaches to evaluation that examine informatics resources from multiple perspectives, using several methodologies, are likely to produce more valuable results • Tailor methods to research questions & stakeholder perspectives • Ensure methodological rigor. See checklists by Johnston et al. & Sullivan & Mitchell for assessment criteria for experimental and non-experimental studies.

26. Step 6: Ensuring that study findings can be generalized • Difficult to achieve in informatics research. Study effects can be context-dependent (‘People don’t use computers: organisations do’) • Qualitative research will focus on small (selected) samples, although may indicate wider issues which could affect generalisability of results • Experimental research may be more generalizable but important to build-in safeguards e.g. increase sample sizes when randomising by practice to correct for intra-cluster correlation.

27. Step 7: Carry out the evaluation study Preparation • Decide whether continuation is justified. • Firm up the methodology • Convince the ethics committee. • Identify & liase with key stakeholders • Consider commercial implications and intellectual property rights.

28. evaluation study continued. • Recruitment • Remember - enthusiasts may not be representative • Design strategies for recruiting patients (& gaining consent) • Detailed study planning • Create written manual of study procedures. (Focuses on the fine detail of who does what at the different stages of the project. May change over time.)

29. evaluation study continued. • Pilot as much of your study procedure as possible • Think carefully about where you intend to do the pilot work. Sites need to be representative of those you intend to use in the main study. Use a small number of test-bed sites to learn of the problems with the resource. • Other issues to consider during the study • Respond immediately to any technical problems or concerns expressed by participants. • Study sites and participants should be kept informed of progress. • Reward participating practices if possible