Grading the evidence in systematic reviews of measurement properties 23 september 2010

1. Grading the evidence in systematic reviews of measurement properties 23 september 2010 Caroline Terwee Knowledgecenter Measurement Instruments Department of Epidemiology and Biostatistics VU University Medical Center

2. What is a systematic review of measurement properties? A review of the measurement properties of all available measurement instruments that aim to measure a particular construct in a particular population. Aim is to select the best instrument for a specific purpose

3. What is special about a systematic review of measurement properties (SR-MP)? • A SR-MP has more than one outcome measure, i.e. multiple measurement properties • Different studies evaluate different measurement properties  the number of studies in the analysis is different per measurement property • The quality of the studies is evaluated per measurement property • Data synthesis is different per measurement property • Evidence for one measurement property may come from different studies • Therefore, a SR-MP is actually a collection of separate reviews per measurement property

4. Methodology of systematic reviews of measurement properties 10 steps

5. Methodology of systematic reviews of measurement properties 10 steps

6. Data synthesis • 2 steps • Decide on combining studies • Homogeneity of study characteristics  study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval) • Methodological quality • Consistency of the results of the measurement properties

7. Data synthesis • 2 steps • Decide on combining studies • Homogeneity of study characteristics  study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval) • Methodological quality • Consistency of the results of the measurement properties • Decide on the analysis: • Quantitative analysis  statistical pooling • Qualitative analysis  best evidence synthesis

8. Example Reliability of the Quebec Pain Disability Scale (QBPDS) Nine studies evaluated reliability; 8 included in the analysis 1. Kopec et al. J Clin Epidemiol 1996 2. Schoppink et al. Phys Ther 1996 3. Fritz et al. Phys Ther 2001 4. Davidson et al. Phys Ther 2002 5. Mousavi et al. Spine 2006 6. De Beer et al. S Afr J Physiother 2008 7. Melikoglu et al. Spine 2009 8. Hicks et al. Pain Med 2009

9. Study population, setting, country

10. Study population, setting, country

11. Methodological quality COSMIN checklist COnsensus-based Standards for the Selection of Health Measurement INstruments Different boxes for each measurement property, with questions regarding quality aspects Box reliability: 14 items to evaluate the quality of a reliability study

12. Methodological quality of the studies

13. Methodological quality of the studies

14. Methodological quality of the studies

15. Consistency of results

16. Results of the studies

17. Results of the studies

18. Step 2: Best evidence synthesis Data syntheses is different per measurement property General guideline: levels of evidence, based on Cochrane Back Review group

19. Example Reliability of the Quebec Pain Disability Scale (QBPDS) Consistent findings of good reliability (ICC>0.70) in three studies of good methodological quality and in four studies of fair methodological quality  Strong evidence for a good reliability (+++)

20. Other measurement properties Internal consistency Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that (sub)scales are unidimensional PLUS Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that Cronbach’s alpha is > 0.70

21. Content validity Strong evidence: all items are considered relevant for the construct, purpose, and target population and the instrument is considered comprehensive. Moderate evidence: the items are considered relevant for the construct or target population and the instrument is considered comprehensive. Limited evidence: only one aspect of content validity is assessed.

22. Construct validity and responsiveness • Levels of evidence as described in the Table are applied. • Challenges • some studies examine more hypotheses than others • some hypotheses are more challenging than others • some comparison instruments are better than others

23. Summary similarities and dissimilarities with GRADE Study limitations (methodological quality) are taken into account, but not to downgrade the level of evidence but to exclude studies from the data analysis Inconsistency is taken into account in applying levels of evidence Indirectness (generalizibility) is usually not taken into account because there are too many differences in study characteristics and the influence on the measurement properties is unclear Impresision (only one study) is very common. This is taken into account in applying levels of evidence Publication bias is not considered, but might be a problem

24. Systematic reviews of measurement properties

25. TODAY 13.45 AULA VU PhD defence Wieneke Mokkink COSMIN checklist COnsensus-based Standards for the Selection of Health Measurement INstruments