Start with the basics: pilot evaluation of eHealth services. Jeremy Wyatt Visiting professor, KIK, AMC Amsterdam Associate director of R&D, NICE, London [email protected] Overview. What is eHealth ? What is evaluation and why do we evaluate ? What to measure in pilot eHealth studies ?
Start with the basics: pilot evaluation of eHealth services
Visiting professor, KIK, AMC Amsterdam
Associate director of R&D, NICE, London
“Using the internet and other electronic media to disseminate or provide access to health and lifestyle information or services”
Cf. Telemedicine – implies a health professional at one or both ends
3. Collect data, analyse results
Aim to generate relevant information to support decisions throughout project
Stakeholders ask questions, evaluators formalise them
Methods depend on question & reliability of answer needed (not on technology):
Challenge: titrating evaluation methods to resources available & reliability of answer required
Who to study: 5-10 typical target users (Nielsen, www.useit.com)
Setting: lab / classroom
NHSDirect Healthspace, www.nhsdirect.nhs.uk
On-screen or paper questionnaires with closed and open ended responses
Introduction, question wording & order, response wording & format will all influence the answers !
Need to pilot instrument, check its reliability (repeatability) & validity (usefulness)
Use published instruments of reasonable reliability & validity, where possible, preserving original wording
Example: TeleMedicine Preparedness Questionnaire, developed to assess preparedness of elderly Americans for virtual home visits by nurses using teleconferencing hardware.
Demiris G et al. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemedicine & Telecare 2000; 6: 278-84
Pilot studies carried out Nov ’02-March ’03 (total consults so far c. 150):
Nurse training needs:
Fit: CES seems a logical replacement for:
Safety & risk exposure
Comments face to face & on form very positive, eg:
Compared to NHSD textphone service, the CES is:
Study produces positive, expected result:
Congratulations all round
Conclude that results can be widely applied
No need to repeat study
Sweep biases, confounders under the carpet
Study produces negative, unexpected result:
Delayed / postponed publication
Conclude that results can never be applied
Repeat study in another setting
Careful search for biases, confounders to explain “anomalous” results
(* to a qualitative researcher, every question is an invitation to participate)
What should we think about a doctor who uses the wrong treatment? Most people would agree that such behaviour was unethical & unacceptable. What, then, should we think about researchers who use the wrong techniques (wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report them selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal.
Huge sums of money are spent annually on research seriously flawed through inappropriate designs, unrepresentative small samples, incorrect analysis and faulty interpretation. Errors are so varied that a whole book on the topic is not comprehensive… We need less research, better research & research done for the right reasons.
The scandal of poor medical research. D G Altman. BMJ 1994; 308: 283-4