Key words: STARD checklist, Cardiology, Diagnostic accuracy

1. Completeness of reporting of studies of diagnostic accuracy in Cardiology journals: a STARD assessment Key words: STARD checklist, Cardiology, Diagnostic accuracy Class 6: Ana Borges, Ana Torres, André Tojal, Célia Soares, Francisca Trigo, Francisco Coutinho, Joana Ribeiro, José Fernandes, Pedro Vasconcelos, Raquel Ferreira, Sílvia Madureira Supervisors: Altamiro da Costa Pereira, MD, PhD; Filipa Almeida, MD Introduction to Medicine; Porto Faculty of Medicine 2008

2. Introduction Objectives Methods Results Conclusion Acknowledgments

3. Introduction Objectives Methods Results Conclusion Acknowledgments • Accurate diagnosis in Cardiology is crucial for clinical intervention and is increasingly important as the number of validated treatment for specific conditions increases. • The use of systematic literature review to inform evidence based practice in diagnostics is rapidly expanding. • Studies of the new diagnostic tests, and their development, are reported in scientific articles. (1) (1) - Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Intern Med.2003; 138:40-44

4. Introduction Acknowledgments Objectives Methods Results Conclusion • Current cardiology practice relies on diagnostic tests using sophisticated technologies that are constantly evolving. Diagnostic accuracy studies are required to validate new diagnostic tests before they are introduced into clinical practice. • These studies typically report sensitivity and specificity, likelihood ratios, diagnostic odds ratio, or area under a receiver operating characteristics (ROC) curve as measures of diagnostic performance. This information enables a clinician to make judgments regarding the potential utility of new tests. (2) (2) Deeks J. Systematic reviews in health care: systematic reviews of diagnostic tests, BMJ 2001; 44:639-650

5. Introduction Acknowledgments Objectives Methods Results Conclusion • A rigorous evaluation process of diagnostic tests before introduction into clinical practice could not only reduce the number of unwanted clinical consequences related to misleading estimates of test accuracy, but also limit health care costs by preventing unnecessary testing. • Several systematic reviews (1995, Reid et al) (3) have emphasized the poor quality of reporting in diagnostic accuracy studies. This poor reporting hampers an adequate judgment of both the internal and the external validity of a study. (3) Scholten RJ, Deville WL, Opstelten W, Bijl D, van der Plas CG, Bouter LM. The accuracy of physical diagnostic tests for assessing meniscal lesions of the knee: a meta-analysis. J Fam Pract. 2001; 50:938–944.

6. Introduction Acknowledgments Objectives Methods Results Conclusion History of the STARD • In 1999, in Rome, the Cochrane Diagnostic and Screening Test Methods Working group discussed the low methodological quality and sub-standard reporting of diagnostic test evaluations, that resulted in the intent of developing a checklist of items that should be included in those reports. • The STARD steering committee started with an extensive search to identify publications on the conduct and reporting of diagnostic studies up to July 2000 .(4) • In January2003, after the discussion based on the search done, we have the first official version of the STARD, which was published simultaneously in eight medical journals. (4) Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003

7. Introduction Acknowledgments Objectives Methods Results Conclusion STARD CHECKLIST • Objective: • The aim of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative is to improve the quality of reporting of studies on diagnostic accuracy. How: • Using a checklist with 25-itemand a flow diagram, which provides information about the method of patient recruitment, the order of test execution and the numbers of patients undergoing the test under evaluation, the reference standard or both. (5) • Advantage: • Complete and accurate reporting allows the reader to detect the potential for bias in the study and to evaluate the general ability of the results. (5) Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA. 1995; 274:645–651.

8. Introduction Objectives Acknowledgments Methods Results Conclusion Importance of our analysis • Cardiology is the area with the most published articles; • This is the first study that analyses how reports on diagnostic accuracy in cardiology comply with the STARD checklist;

9. Introduction Acknowledgments Objectives Methods Results Conclusion Main objective To assess the quality of reporting of studies on diagnostic tests in cardiology journals, based on the STARD checklist. Secondary objectives • To analyse the quality of diagnostic testsstudies in the course of the years; • To compare the differences between the articles published before and after the creation of the STARD checklist; • To evaluate the influence of the journals’ impact factor on the quality of the articles which report diagnostic tests in cardiology journals.

10. Introduction Acknowledgments Objectives Methods Results Conclusion Standard query to find articles that report studies of diagnostic accuracy (in PubMed) Articles’ search Articles’ selection Articles’ evaluation Statistical analyze (((((((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields)) OR (((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All fields]) OR "roc and"[All fields]) OR "roc area"[All fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields])) OR "likelihood ratio"[All Fields]) Devillé W. Conducting systematic reviews of diagnostic studies; didactic guidelines. Medical research methodology. 2002

11. Introduction Acknowledgments Objectives Methods Results Conclusion Search the journals Included all journals classified as “cardiac and cardiovascular systems” in the ISI web of knowledge except those who are not found on pubmed. Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Included all 71 journals found on Pubmed

12. Methods Introduction Objectives Results Conclusion Acknowledgments Final query (after addition of terms to specify the search to the journals) (((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields])) AND (Circulation [Journal] OR Circulation Research [Journal] OR Journal Of The American College Of Cardiology [Journal] OR European Heart Journal [Journal] OR Cardiovascular Research [Journal] OR Journal Of Molecular And Cellular Cardiology [Journal] OR American Journal Of Physiology-Heart And Circulatory Physiology [Journal] OR Journal Of Thoracic And Cardiovascular Surgery [Journal] OR Trends In Cardiovascular Medicine [Journal] OR American Journal Of Cardiology [Journal] OR Journal Of Heart And Lung Transplantation [Journal] OR Chest [Journal] OR American Heart Journal [Journal] OR Journal Of Cardiovascular Pharmacology [Journal] OR Progress In Cardiovascular Diseases [Journal] OR Journal Of Cardiovascular Magnetic Resonance [Journal] OR Heart [Journal] OR Journal Of Nuclear Cardiology [Journal] OR Annals Of Thoracic Surgery [Journal] OR Journal Of The American Society Of Echocardiography [Journal] OR Current Opinion In Cardiology [Journal] OR Current Problems In Cardiology [Journal] OR Basic Research In Cardiology [Journal] OR Catheterization And Cardiovascular Interventions [Journal] OR Respiratory Medicine [Journal] OR Canadian Journal Of Cardiology [Journal] OR European Journal Of Cardio-Thoracic Surgery [Journal] OR European Journal Of Heart Failure [Journal] OR Journal Of Heart Valve Disease [Journal] OR Clinical Cardiology [Journal] OR Cardiovascular And Interventional Radiology [Journal] OR International Journal Of Cardiology [Journal] OR Herz [Journal] OR Cardiovascular Drugs And Therapy [Journal] OR Journal Of Cardiothoracic And Vascular Anesthesia [Journal] OR Cardiovascular Drug Reviews [Journal] OR Zeitschrift Fur Kardiologie [Journal] OR Pediatric Cardiology [Journal] OR Cardiovascular Surgery [Journal] OR Thoracic And Cardiovascular Surgeon [Journal] OR Journal Of Interventional Cardiac Electrophysiology [Journal] OR Revista Espanola De Cardiologia [Journal] OR Cardiology [Journal] OR Heart & Lung [Journal] OR Cardiology In The Young [Journal] OR Heart And Vessels [Journal] OR Journal Of Electrocardiology [Journal] OR Journal Of Cardiovascular Surgery [Journal] OR International Journal Of Cardiac Imaging [Journal] OR Texas Heart Institute Journal [Journal] OR Scandinavian Cardiovascular Journal [Journal] OR ActaCardiologica [Journal] OR Europace [Journal] OR Archives Des Maladies Du Coeur Et Des Vaisseaux [Journal] OR Cardiovascular Pathology [Journal] OR Japanese Heart Journal [Journal] OR Journal Of Cardiac Surgery [Journal] OR Kardiologiya [Journal]) Articles’ search Articles’ selection Articles’ evaluation Statistical analyze

13. Introduction Acknowledgments Objectives Methods Results Conclusion From all articles retrieved by the query Articles’ search Articles’ selection Articles’ evaluation Statistical analyze • Random sample of 100 articles stratified by years • Sample of 10 articles per year • Years included: • - Until 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007.

14. Methods Introduction Acknowledgments Objectives Results Conclusion Reading and analyzing the abstract Articles’ search Articles’ selection Articles’ evaluation Statistical analyze The abstract is read and analyzed with the inclusion/exclusion criteria Inclusion - to report a study of diagnostic accuracy - the diagnostic test(s) is (are) used in cardiology - the full text is available

15. Methods Introduction Acknowledgments Objectives Results Conclusion Reading and analyzing the abstract Articles’ search Articles’ selection Articles’ evaluation Statistical analyze • Exclusion • Is a clinical trial, a letter, an editorial, a practice guideline, a meta-analysis or a review • -The article could not be found in the Internet, the facilities of the School of Medicine or the local libraries • -The article is written in languages other than English, Portuguese, French or Spanish • - The article found is not complete

16. Introduction Acknowledgments Objectives Methods Results Conclusion Revision Articles’ search Articles’ selection Articles’ evaluation Statistical analyze If excluded: - the reason of exclusion is registered If doubted: - the abstract is reanalyzed by other selector When disagreement or doubt occur, the article will be discussed by both selectors. When no decision could be achieved, a third selector will be consulted When this situation occurs, PMID of each opened article will be registered

17. Introduction Acknowledgments Objectives Methods Results Conclusion Final samples 2007 → 139 articles Articles’ search Articles’ selection Articles’ evaluation Statistical analyze 2006 → 136 articles 2005 → 125 articles 2004 → 132 articles 2003 → 125 articles 2002 → 147 articles 2001 → 112 articles 2000 → 172 articles 1999 → 112 articles Until 1998 → 1585 articles

18. Introduction Acknowledgments Objectives Methods Results Conclusion Final samples Articles’ search Articles’ selection Articles’ evaluation Statistical analyze TOTAL → 2785 articles

19. Introduction Objectives Methods Results Acknowledgments Flowchart Articles’ search Articles’ selection Articles’ evaluation Statistical analyze

20. Introduction Acknowledgments Objectives Methods Results Conclusion Gantt Chart Articles’ search Articles’ selection Articles’ evaluation Statistical analyze

21. Objectives Methods Results STARD CHECKLIST Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Each item values 1 point Several criteria was defined for the STAR application

22. Objectives Methods Results STARD CHECKLIST Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Each item values 1 point Several criteria was defined for the STAR application

23. Introduction Acknowledgments Objectives Methods Results Conclusion Evaluators – Distribution for year Articles’ search Articles’ selection Articles’ evaluation Statistical analyze Article’s components - Title, - Abstract, - Keywords - Introduction - Methods - Results - Discussion One evaluator for each year Every article´s components are seen for the same evaluator 10 articles for each evaluator

24. Introduction Acknowledgments Objectives Methods Results Conclusion Points to analyse: Articles’ search Articles’ selection Articles’ evaluation Statistical analyze • Final score of each article • Average score of the articles for each group (year of publication, impact factor) • Interobserver variability • The status of each STARD item in each year

25. Methods Introduction Acknowledgments Objectives Results Conclusion Statistical Analyse Articles’ search Articles’ selection Articles’ evaluation Statistical analyze

26. Introduction Acknowledgments Objectives Methods Results Conclusion Final Selection States FromArticles’ Selection FromArticles’ Evaluation

27. Introduction Acknowledgments Objectives Methods Results Conclusion Reasons for the Exclusion FromArticles’ Selection FromArticles’ Evaluation

28. Introduction Acknowledgments Objectives Methods Results Conclusion Final Classification of the Items FromArticles’ Selection FromArticles’ Evaluation

29. Introduction Acknowledgments Objectives Methods Results Conclusion ICC for each Item of the List FromArticles’ Selection FromArticles’ Evaluation

30. Introduction Acknowledgments Objectives Methods Results Conclusion Evolution of the Articles’ Quality FromArticles’ Selection FromArticles’ Evaluation

31. Introduction Acknowledgments Objectives Methods Results Conclusion The Impact Factor of the Journal FromArticles’ Selection FromArticles’ Evaluation

32. Introduction Acknowledgments Objectives Methods Results Conclusion AbouttheSTARD CHECKLIST – Evaluators’ pointofview . Some items of evaluation are rigorously asked for in a section, but in the articles, several times, they appear in another one; . Because some STARD items are potentially affected by the subjective judgment of the reviewers due to differences in interpretation of the STARD, disagreement arises;

33. Introduction Acknowledgments Objectives Methods Results Conclusion Relation Between the Articles’ Score and the Impact Factor of the Journal FromArticles’ Selection FromArticles’ Evaluation

34. Introduction Acknowledgments Objectives Methods Results Conclusion AbouttheSTARD CHECKLIST – Evaluators’ pointofview . The data of the participants (age, sex, symptoms,…) and their dates of recruitment should be included in the methods; . The list has 25 items, but it’s possible each one can have more then one aspect to evaluate, which creates confusion in the classification to attribute.

35. Results Introduction Acknowledgments Objectives Methods Conclusion Expected Results vs Obtained Results • Expected • As we can see in other medical areas where this project had been done, such as obstetrics and gynecology and others, we expect that this checklist had had impact on the articles written after it as well as that they had improve its qualities due to STARD Checklist.

36. Introduction Acknowledgments Objectives Methods Results Conclusion Expected Results vs Obtained Results Obtained . The differences found in the articles were nothing to do with variances observed between groups, we mean the year of publish is not related with the improvement of the articles (p<0,005). . Therefore we assume that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy and so we suppose that the improve in this kind of articles is due to the progress of knowledge along the years and not related with the creation of this checklist, as shown in the graphic. . We come to conclusion that there is no correlation between them, since we reach a p value equal to 0,210, that is not significant and by the way that we can declare that the year of publish or the improve verify in the articles does not interfere with the impact factor of the journals.

37. Introduction Acknowledgments Objectives Methods Results Conclusion Conclusion Our study shows that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy .

38. Introduction Objectives Methods Results Acknowledgments Acknowledgments • Altamiro da Costa Pereira, MD,PhD • Filipa Almeida, MD, PhD • Cristina Santos, MD, PhD • Luís Azevedo, MD, PhD