Are the results valid?

1. Are the results valid? • Was the validity of the included studies appraised?

2. Are the results valid? • Was the validity of the included studies appraised? • When conducting a meta-analysis the overall results are weighted by the size of the contributing studies. However further weighting may be given to the most rigorously conducted studies. This can be achieved by making a judgement on the validity of the studies. This may be qualitative (a comment about key characteristics of the study) or quantitative where each study is scored say out of 10, and the results of the study are then weighted according to the score achieved.

3. Give an example of the Oxford scoring system.

4. Are the results valid? • Were the assessments of the included studies reproducible?

5. Were the assessments of the included studies reproducible? • If the quality of the studies have been assessed then would the reader be able to understand the scoring system used and apply it to the papers? • Some meta-analyses may appraise the studies individually and give a score out of ten, but not make transparent why a paper scored badly - was it because there were a lot of drop outs from the study,or because the randomisation had been poorly done etc.

6. Are the results valid? • Were the results consistent from study to study?

7. Combinability • First of all one should ask whether the results of separate trials can be meaningfully combined? • What criteria were used to decide that studies were similar enough to be combined?? • Issues include study designs, types of patients and outcomes E.g. Can we combine RCTs with observational studies, or studies using cotrimoxazole with those using trimethoprim?

8. Were the results consistent from study to study? • Each study is considered to be from a different population, the rate varies from study to study and differences can be because of experimental error, chance or differences in the populations. There are tests for homogeneity (or heterogeneity) which help decide the degree of caution in interpreting or pooling the results.

9. Were the results consistent from study to study? • In general differences in treatments are likely to differ in magnitude rather than direction and this can be displayed graphically to support or refute homogeneity between studies. Results are homogeneous if they reflect the same “true” effect. • References • Tests for homogeneity of effect in epidemiologic investigation. American Journal of Epidemiology 1977; 106: 125-9 • Meta-analysis in clinical research. Annals of Internal Medicine 1987; 107: 224-33

10. Add in some diagrams showing studies with heterogeneity and some that are homogeneous

11. II. What are the results? 1. What are the overall results of the study? • Look at the Relative Risk (RR) of the main outcome in the two groups. In a meta analysis this is usually presented as a diamond with the 95% confidence intervals grouped about a line indicating the null hypothesis. • A number of outcomes may be considered at once so more than one diamond may be presented.

12. Meta-analysis PETO GRAPH Outcome A Study A B C D E Total 0 Placebo Drug

13. II. What are the results? 1. What are the overall results of the study? • Look at the Relative Risk (RR) of the main outcome in the two groups. • What about sub-group analyses? I.e. what are the results of the individual studies. Do some studies stand out as being different. Have the log odds ratios been calculated or has the meta analysis managed to pool original data in which case other sub group analyses may be possible - one of the advantages of pooling the original data.

14. II. What are the results? 1. What are the overall results of the study? • Look at the Relative Risk (RR) of the main outcome in the two groups. • What about sub-group analyses? • Can you calculate the Number Needed to Treat (NNT) from the results presented? This necessitates an estimate of the absolute risk in the treated and control groups rather than simply having the odds ratio presented.

15. II. What are the results? 2. How precise are the results?

16. II. What are the results? • How precise are the results? • A meta analysis should always present the 95% confidence intervals for each result.

17. III. Will the results help me in caring for my patients? 1. Can the results be applied to my patient care?

18. Will the results help me in caring for my patients? 1. Can the results be applied to my patient care? Clinical significance. • Refer back to the clinical problem • Are the studies generalisable to our patient? • Age, ethnicity, community or hospital patients etc?

19. Generalisability • Major problem with RCTs - i.e. do the results of this study apply to my patients? • Need to know details of patients included in all the studies. This will help decision about the generalisability of the results and also may reveal reasons for heterogeneity

20. Will the results help me in caring for my patients? 2. Were all the clinically relevant outcomes considered?

21. Will the results help me in caring for my patients? 2. Were all the clinically relevant outcomes considered? • What about other outcomes - particularly harm. • What about quality of life issues?

22. Will the results help me in caring for my patients? 3. Are the benefits worth the harms and costs?

23. Will the results help me in caring for my patients? 3. Are the benefits worth the harms and costs? • Cost differences in treatments. • Greater benefits and less side effects?

24. Critical Appraisal of a meta analysis - methodology • Selection bias • Generalisability • Combinability • Consistency • Statistics • Sensitivity analysis

25. Statistics • Pool the log odds ratios • Standardised average • Regression analysis on pooled data • Sub group analysis

26. Sensitivity analysis • Can be done using quality score of trials e.g. published vs unpublished, observational and RCTs, details of randomisation • Publication bias - i.e. how many studies showing no difference would have to exist but not published to invalidate my results?