Evaluation criteria in practice development using mobile ICT

1. Evaluation criteria in practice development using mobile ICT • Gunvor Gard och Anita Melander-Wikman • Dept of Health Sciences • Luleå University of Technology • Gunvor.Gard@ltu.se • Anita.Melander-Wikman@ltu.se

2. Technologies and services for elderly • Increased number of elderly in Nordic countries • Increased demand on prevention and rehabilitation services. • Mobility? Well-being? Quality of life? • Utility - functionality? Usability testing? • Evaluation methods – scientific demands?

3. Prevention • Primary, secondary,tertiary • Interventions on different organizational levels: • Political,economic,cultural • Work organizational • Group • Individual

4. Scientific demands? Clinical relevance? Technical relevance? Functional relevance? • Evidence –the use of best scientific criteria to develop the best practice • Level 1 – meta analysis of RCT • Level 4 consensus between professionals in the field (Turner-Stokes,2002)

5. Sackets rules of scientific evidence,1990 • Level 1. Large randomized controlled trials • Level 2. Small randomized controlled trial • Level 3. Non-randomized concurrent cohort comparisons • Level 4. Non-randomized historical cohort comparisons • Level 5. Case series without controls

6. Standards should be evidence-based • Different levels of evidence • Multiprofessional cooperation step-wise process • Technologies in a prototyping phase • Can the new technology change an organizational structure or practice • No consensus about what is the best way to evaluate new systems or technologies

7. E-HomeHealthCare@NorthCalotte • Coordinated trials in 5 municipalities in 3 countries • Aim: Improve information exchange in homecare services by mobile ICT equipment • Methods: Questionnaires,interviews, focusgroup interviews,observations • Theories of participatory action research • Marratech web-based conference tool was used

8. Which criteria are realistic and relevant to use? • Criteria based on perceived needs and priorities • Criteria based on objective data • Criteria based on available resources and existing policies

9. Criteria based on perceived needs and priorities • Arbitary criteria from individuals or groups in a position of authority • Experiential criteria – a community’s perceived needs and priorities,know-how • Utility criteria –focused on local utility aspects

10. Criteria based on objective data • Historical criteria –based on previous performance data and trend analyses • Normative criteria – for example national average • Scientific criteria . Outcomes in controlled studies such as RCT and metaanalysis towards a best practice

11. Criteria based on available resources and existing policies • Propriety standards – legal ethical norms • Feasibility standards – cost-effectiveness,political and practical norms

12. Which criterias were considered in the EHHC project? • Criteria based on perceived needs and priorities • Criteria based on objective data • Criteria based on available resources and existing policies • Experiential criteria – a community’s perceived needs and priorities,know-how • Utility criteria –focused on local utility aspects • Scientific criteria .

13. Develop the best approach • Evidence – to use all possible criteria as best available evidence • No methodological golden standard exists so far • Use already reliable and valid methods or test your method before use • Comparisons possible if we use the same methods

14. Development of a questionaire • Systematic literature search in relation to the research question – key words • Identification of relevant dimensions for questions • Development of questions – item generation • Factor analysis of the questionnaire – item reduction • To limit the questionnaire and see if the literature dimensions can be identified in the structure • Dimensions confirmed or not? • Reliability and Validity testing

15. Evaluation criteria in practice development using mobile ICT • Focus on participation as well as knowledge-gathering • Focus on scientific multi-method development • Focus on multiprofessional cooperation • Participatory action research can be used in practice development using mobile ICT