A Critical Look at Diagnostic Criteria : Time for a Change ?

1. A CriticalLook at DiagnosticCriteria: Time for a Change? Hasan Yazici IstanbulUniversity

3. Plan • A brokenankle • Issueswith ISBD (Behçet) criteria • The EULAR/ACR criteriafor RA: an issue in validation • Thoughtbarriers • Perhaps…

4. Ottowa AnkleRules sensitivity~ 100%, specificity ~ 40% IG Stiell et al. AnnEmergMed, 1992

5. The Ottowa AnkleRules • Theywere not directlyaboutdiagnosing, theywereaboutwhenweshouldget a radiographtohelp us diagnose an anklefracture in the relevant setting. • Theyavoidedmanyunnecessaryradiographs but allnecessaryradiographsweretaken.

6. BlinkThePower of ThinkingWithoutThinking TheGettyKouros MalcolmGladwell (2005)

7. from: Blink(M. Gladwell ) A computer-derived protocol to aid in the diagnosis of emergency room patients with acute chest pain. L. Goldman et al. N Eng J Med, 1982 To determine whether data available to physicians in the emergency room can accurately identify which patients with acute chest pain are having myocardial infarctions, we analyzed 482 patients at one hospital. Using recursive partitioning analysis, we constructed a decision protocol in the format of a simple flow chart to identify infarction on the basis of nine clinical factors. In prospective testing on 468 other patients at a second hospital, the protocol performed as well as the physicians. Moreover, an integration of the protocol with the physicians' judgments resulted in a classification system that preserved sensitivity for detecting infarctions, significantly improved the specificity (from 67 per cent to 77 per cent, P less than 0.01) and positive predictive value (from 34 per cent to 42 per cent, P = 0.016) of admission to an intensive-care area. The protocol identified a subgroup of 107 patients among whom only 5 per cent had infarctions and for whom admission to non-intensive-care areas might be appropriate. This decision protocol warrants further wide-scale prospective testing but is not ready for routine clinical use

8. MI prediction rules L Goldman et al. N Engl J Med, 1982 • 1. Unstable angina • 2. Basilar rales • 3. BP<10 mmHg

9. TheCookCounty MI PredictionRules • Theywere not directlyaboutdiagnosing, theywereaboutwhenweshouldsend a patientto CCU whenwesuspected MI in therelevantsetting. • Theyavoidedmanyunnecessaryadmissionsto CCU but thosepatientswhoreallyneededthe CCU gotthere.

10. EULAR/ACR VasculitisCriteriaGroup(2008) “PerhapsthemostrobustcriteriapertaintoBehçet’sdisease, whereinternationalcolloboration has ledto a validatedproposaleffectiveforbothclinicalandresearchpurposes.” “Thechallange of producingvalidateddiagnosticcriteriafortheseheterogeneousdiseasesthataresuitableforuse in clinicalresearch as a classificationtool, is a formidableone. However, as theinternationalcommunity has examplifiedwithBehçet’sdisease, it is achievable.” 10

11. Plan • A brokenankle • Issueswith ISBD (Behçet) criteria • The EULAR/ACR criteriafor RA: an issue in validation • Thoughtbarriers • Perhaps

12. InternationalStudyGroupDiagnosticCriteria(Lancet, 1990) Oralulcers (~100%) + Two of below: Genitalulcers(80%) Skin lesions(80%) Eyelesions(50 %) Pathergy (50 %) 12

13. Hunder GG. TheUseandMisuse of ClassificationandDiagnosticCriteriaforComplexDiseases. AnnInternMed , 1998 13

14. InternationalStudyGroupDiagnosticCriteria(Lancet, 1990) Oralulcers (~100%) + Two of below: Genitalulcers(80%) Skin lesions(80%) Eyelesions(50 %) Pathergy (50 %) Classification 14

15. InternationalStudyGroupDiagnosticCriteria(Lancet, 1990) Oralulcers (~100%) + Two of below: Genitalulcers(80%) Skin lesions(80%) Eyelesions(50 %) Pathergy (50 %) Classification ? 15

16. InternationalStudyGroupDiagnosticCriteria(Lancet, 1990) Oralulcers (~100%) + Two of below: Genitalulcers(80%) Skin lesions(80%) Eyelesions(50 %) Pathergy (50 %) Classification & Diagnostic 16

17. ISBD CriteriaforBehçet’sDisease(Lancet, 1990) 17 17

18. ISBD CriteriaforBehçet’sDisease(Lancet, 1990) L Goldman et al. N Engl J Med 1982 18 18

19. ISBD Criteria-Methods I • Information on 914 patientswithdiagnosed BS werecollectedfrom 12 centers. • Of these, theclinicalfeatures of onlythosewith oral ulcers (n= 887) werecomparedwiththoseamong 97 patientswithotherinflammatorydiseasesand oral ulcers. • Thecriteriawereinitiallyformulatedamong a 60% sample of the BS poolandthecriteriawerelatervalidatedamongthe 40% sample.

20. ISBD Criteria-Methods II • Theexpectedweight of evidences (Turing) calculatedforeachdiseasefeature in thetrainingsample. • Fromthesethenewdiseasecriteriawereformed. • Thesewerevalidatedamongthevalidationsample (therandomlychosen 40% sample)

21. ISBD Criteria-Methods II • Theexpectedweight of evidences (Turing) calculatedforeachdiseasefeature in thetrainingsample. • Fromthesethenewdiseasecriteriawereformed. • Thesewerevalidatedamongthevalidationsample (therandomlychosen40% sample)

22. 22 22

23. 23 23

24. TheMeaning of Fulfillingthe ISBD Criteria(sensitivity = 90%; specificity=95%) Criteria + (+LR) : 0.90/1-0.95 = 18 x pre-test odds Criteria - (-LR) : 1 - 0.90 /0.95 = 0.10 x pre-test odds 24

25. TheMeaning of Fulfillingthe ISBD Criteria(sensitivity = 90%; specificity=95%) Criteria + (+ LR) : 0.90/1-0.95 = 18 x pre-test odds Criteria - (-LR) : 1 - 0.90 /0.95 = 0.10 x pre-test odds 25

26. Issueswiththe ISBD criteria • Did the comparator group represent conditions thatusually come into the differential diagnosis of BS? • The trouble with the validation group: - randomly chosen from the initial set • The trouble with the control group: - lack of patients with CNS or major vessel disease • What will be the pre-test odds of having BS in the setting in which these criteria will be used? • What will be the pre-test oddsof having one of the comparator diseases in the setting in which these criteria will be used? • Whatwill be thepre-test odds of having yet undefineddiseases in thesettingthesecriteriawill be used?

27. Plan • A brokenankle • Issueswith ISBD (Behçet) criteria • The EULAR/ACR criteriafor RA: an issue in validation • Thoughtbarriers • Perhaps

34. Validation vs.Total Group Total includes data fromthevalidationgroupsincludes data from

35. Inbrief: • Wehaveproblems in makinggoodvalidationgroupsforourclassification/diagnosticcriteria. • Perhapsthis is duetothenearimpossibility of formulatingsuchcontrolgroups.

36. Plan • A brokenankle • Issueswith ISBD (Behçet) criteria • The EULAR/ACR criteriafor RA: an issue in validation • Thoughtbarriers • Perhaps…

37. Thought Barriers I. Themisconception of tryingtoavoidcircularity in criteriamaking II. Promise of an universaldiagnosticcriteria, somehowdifferentfromclassificationcriteria, tocome III. A lack of appreciation of preand post test (criteria) odds IV. A lack of appreciation that we begin to diagnose from a certain symptom or sign V. A lack of consideration of whether we have to diagnose and when we do what do we plan to do with our diagnosis VI. A lack of appreciationthattherearediseasesthatare yet to be defined. VII. A lack of considerationthatweshouldperhapsinvolveourpatients in namingandwhatwe plan to do withtheirdiagnosis (es)

38. Thought Barriers I. Themisconception of tryingtoavoidcircularity in criteriamaking II. Promise of an universaldiagnosticcriteria, somehowdifferentfromclassificationcriteria, tocome III. A lack of appreciation of preand post test (criteria) odds IV. A lack of appreciation that we begin to diagnose from a certain symptom or sign V. A lack of consideration of whether we have to diagnose and when we do what do we plan to do with our diagnosis VI. A lack of appreciationthattherearediseasesthatare yet to be defined. VII. A lack of considerationthatweshouldperhapsinvolveourpatients in namingandwhatwe plan to do withtheirdiagnosis (es)

39. CircularReasoning • Alldiagnoses/classifications - evenwhenspecifichistology/microbiologyareinvolved - arebased on definitionsandthushavesomecircularity. • Howevercircularreasoning is onlypresentwhentheconclusion is nothingmorethan a reiteration of thepremise (s) of thereasoning.

40. CircularReasoningPresent • Ourrheumatologyunitdoes not make a diagnosis of RA unless 3 monthspassaftertheonset of symptoms. • Among 300 patientswithinflammatoryarthritisseen in ourclinicwithin 3 months, thearthritiswentawaybysymptomatictherapy in 120 patients. • Itwasinterestingtonotethattherewere no patientswith RA amongthese 120 patients.

41. CircularReasoningAbsent • Ourunitdefines RA as symmetricalpolyarthritis of unknowncausewhichlasts at least 3 monthsandinvolves at least 2/3 jointgroups, consisting of MCP’s, wristsorMTP’s. • Among 1000 newpatientsseen in ourclinicwithin a year 490 satisfiedthisdefinition. • Almost a half of thenewpatientswesee in a year has RA.

42. Inbrief: • Comingto a conclusionunawarethattheconclusionreachedwasinescapable is “circularreasoning”. • Tosearchforandfindwhat has beendefined is NOT circularreasoning.

43. JF FriesArch Intern Med, 1984

44. JF FriesArch Intern Med, 1984

45. Thought Barriers I. Themisconception of tryingtoavoidcircularity in criteriamaking II. Promise of an universaldiagnosticcriteria, somehowdifferentfromclassificationcriteria, tocome III. A lack of appreciation of preand post test (criteria) odds IV. A lack of appreciationthatwebegintodiagnosefrom a certainsymptomorsign V. A lack of consideration of whether we have to diagnose and when we do what do we plan to do with our diagnosis VI. A lack of appreciationthattherearediseasesthatare yet to be defined. VII. A lack of considerationthatweshouldperhapsinvolveourpatients in namingandwhatwe plan to do withtheirdiagnosis (es)

46. The promise of diagnostic criteria to come

47. JF Fries et al. Arthritis Rheum, 1994

48. JF Fries et al. Arthritis Rheum, 1994

49. Why Not an UniversalDiagnosis/ClassificationScheme ?

50. WhyNot an UniversalDiagnosis/ClassificationScheme ? • Therearedifferentreasonswhywediagnoseorclassify. • Therearedifferences in diseasepresentationdepending on: • Frequency • Subspecialty • Geography • Diseasecourse • Yet undefineddiseases