15 de Abril de 2008

1. Reproducibility of QUOROM Checklist: Are Meta-Analyses in Good Hands? Introdução à Medicina II Turma 17 15 de Abril de 2008

2. A Meta-Analysis is a review in which bias has been reduced by the systematic identification, appraisal, synthesis and statistical aggregation of all relevant studies on a specific topic according to a predetermined and explicit method. Systematic Review Overview Meta-Analysis Introduction What is a Meta-Analysis?

3. In 1987, a survey showed that only 24, out of 86 English-language meta-analyses, reported all the six areas considered important to be part of a meta-analysis: In 1987 Study Design Control of Bias Combinality Statistical Analysis Sensitivity Analysis Application of Results

4. In 1992 this survey was updated with 78 meta-analyses and the researchers noted that methodology has definitely improved since their first survey; However it needed better searches of the: Literature; Quality evaluations of trials; Synthesis of the results. In 1992

5. So, in 1999, several researchers suggested and created the Quality of Reporting of Meta-Analyses (QUOROM) Statement to improve and standardise reporting. The QUOROM Statement – that includes a checklist and a trialflowdiagram – describes the preferred way to present the different sections of a report of a Meta-Analysis; It is organized into 21 headings andsubheadings. In 1999

6. The number of published meta-analyses has increased over time. According to a study, after the QUOROM statement theestimated mean quality score of the reports increased from 2.8(95% CI; 2.3–3.2) to 3.7 (95% CI; 3.3–4.1),that represented an estimated improvement of 0.96 (95% CI; 0.4–1.6, p = 0.0018 two sided t-test). However, the QUOROM group admits itself that this checklist requires continuous research in order to improve the quality of a meta-analysis. QUOROM

7. Reproducibility But what is Reproducibility? Why is it so important? Reproducibility is one of the main principles of the scientific method, which refers to the ability of a test or experiment to be accurately reproduced by someone else working independently.

8. Reproducibility • The lack of reproducibility can lead to major consequences: • a failure in the reproducibility will most probably end in results' heterogeneity; • at a clinical level, if a diagnostic test is not reproducible there is the risk of a patient being wrongly diagnosed; • non-reproducible items of a checklist can lead to a decrease on its credibility and, consequently, of the meta-analyses that used it as a model.

9. Aims The question we want to answer is if the QUOROMChecklist is a reproducible method in the evaluation of Meta-Analysis. Primary Aim: Evaluate the reproducibility degree of the QUOROM Checklist

10. Aims Secondary Aims: Specify which points of the QUOROM Checklist are less reproducible; Verify if there are differences in the reproducibility between the evaluation of meta-analysis from Low Impact Factor journals and from High Impact Factor ones.

11. Methods - Selection of Studies Our target population was the meta-analyses. We had to select a considerable sample of meta-analyses, so we decided to select a total of 52. Our inclusion criteria were: • The article being published in a medicine subjects’ journal; • The article being published in a journal with impact factor ≤2 or ≥8; • The article reporting a meta-analysis; • The article being published in the last three years (2005-2008); • Having access to online full text.

12. Low IF Journals High IF Journals Methods - Selection of Studies First, we separated 40 journals using a Stratified Sampling Method. From Journals of ISI Web of Knowledge that fit our criteria (n=1234), we selected: 20 Journals 20 Journals IF ≥ 8 (82 journals) 0 < IF ≤ 2 (1234 journals) IF – Impact Factor

13. We repeated the whole process of selection of the articles until we had enough meta-analyses. 26 26 Methods - Selection of Studies Low IF Journals: 48 meta-analyses High IF Journals: 219 meta-analyses After this, we proceeded to the selection of the Meta-Analyses. For that, we used a Multi-Stage Sampling Method. The totality of the Journals’ articles were removed from each stratum, following the inclusion criteria previously described. Pool n.2 Pool n.1 High IF Meta-Analyses Low IF Meta-Analyses

14. Methods - Selection of Studies The impact factor of the journal from where each Meta-Analysis came, the name of the journal, the authors and the year of publication were recorded in a database, which was kept secret until the evaluation of the checklist was concluded. It was used only at the end to find out if Reproducibility and Impact Factor were related. Pool n.2 Pool n.1 High IF Meta-Analyses Low IF Meta-Analyses

15. 26 26 52 Methods - Selection of Studies Pool n.1 Pool n.2 Low IF Meta-Analyses High IF Meta-Analyses Finally, we mixed all the articles in a single pool, occulting the strata from each one came. Pool n.3 52 Meta-Analyses

16. Methods - Study Procedures • Before analyzing we established some rules that helped us understanding each item of the checklist: • If a certain item was present in the meta-analysis, but not in the place the checklist determines, we would not consider the item present; • When a item had more than one point, we would only consider it present if the meta-analysis answered to more than half of the points;

17. Methods - Study Procedures • At the item (e), we would give more importance to the point that ensures the replication of the methods; • At the item (o), the meta-analysis had to have a diagram describing trial flow, so that the item could be considered.

18. By analyzing a meta-analysis, each student had to insert the data in the SPSS program. For each item, number 1 was attributed to those which are covered in the meta-analyses, and number 0 to those which aren’t. Methods - Study Procedures Each student/investigator analysed a group of 4 articles and submitted them to the QUOROM Checklist. After the students’ analysis, the articles were mixed again. Then, each student analysed another 4 articles, randomly selected from the 48 articles previously analysed by the rest of the group. This way, each student/investigator analysed different articles.

19. Methods - Study Procedures Thus, our study can be classified as an observational, cross sectional study, whose methods are characteristic of a survey study, and whose purpose is to study the reproducibility.

20. Methods - Variables Description Our variables are: • The actualImpact Factor of the journals from which we randomly selected the articles; • The year of publication of the articles; • The Impact Factor of the journals from which we randomly selected the articles at the year of publication; • The classification of each item of the checklist: we considered thirty-six categorical variables, which can have two numerical codes: 1 or 0. These are our expected outcome of research.

21. Methods - Variables Description From the classification of the items we had other variables: • Summation of the present items by observer 1; • Summation of the present items by observer 2; • Average of the two summations; • Difference between the summations; • Number of concordances between the two observers by article.

22. Methods - Statistical Analysis • Concordance in each Item of the Checklist • (reproducibility of each Item) • We made eighteen concordance tables to calculate: • The proportion of concordance and 95% confidence intervals*; • Positive proportion of concordance; • Negative proportion of concordance; • Kappa Factor. • * we used a normal distribution but with those whose limit of confidence intervals was over one, we used a binomial distribution.

23. Results

24. Methods - Statistical Analysis • Global Reproducibility • The comparison of the summation of each observer was done using the ICC method (Intraclass Correlation Coefficient). • Then we represented the concordance limits of the “difference between the summations” in a scatterplot: • For that, we had to be sure that this variable followed a normal distribution and, if so, to calculate the mean and the standard deviation, all this by making an histogram.

25. Results • ICC = 0,729 • The ICC method revealed that 72,9% of the total variance is explained by the variance between the articles.

26. Results • Histogram: differences between the summations • Following a normal distribution, it would be expected that the mean was 0, so it may have occurred a systematic error in the study. • The concordance limits were [- 4,934 ; 4,434] • This means that 95% of the differences between the summations are in this interval.

27. Methods - Statistical Analysis • Relation between IF and Reproducibility • For this analysis we didn’t use the actual impact factor, but the one at the year of publication of the articles*. • We made two scatterplots, to see if there were correlation between: • The “difference between the summations” and impact factor; • The “number of concordances between the two observers by article” and the impact factor. • * As the ISI Web of Knowledge database wasn’t updated with the impact factors of 2007, in the articles published in that year we used the impact factor of 2006.

28. No correlation between “impact factor” and “difference between the summations” were found, nor between impact factor and “number of concordances between the two observers by article”, because in both scatterplots [figures 4, 5] there was not any preferential orientation of the points Results