Grading of Recommendations Assessment, Development, and Evaluation GRADE Working Group

1. Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group www.gradeworkinggroup.org

2. Acknowledgement ~Prof. Patrick M Bossuyt and Prof. Victor M. Montori

3. Which tests should be recommended, and with what strength? Why grade recommendations? A systematic and explicit approach to making judgments about the quality of evidence and the strength of recommendations can help to prevent errors, facilitate critical appraisal of these judgments, and can help to improve communication of this information. Recommendations about Diagnosis

4. Grading Recommendations: Strong or Weak (For or Against) Strong recommendations strong methods accurate test impacts outcome few downsides of testing strategy indicating a judgment that a majority of well informed people will make the same choice (high confidence, low uncertainty) expect non-variant clinician and patient behavior - most patients should receive the intervention diminished role for clinical expertise - focus on implementation & barriers focused role of patient values and preferences - emphasis on compliance and barriers could be used as a performance / quality indicator decision aids not likely to be needed medical practice is expected to not to vary much

5. Grading the Evidence: Evaluating Diagnostic Studies Evidence concepts scientific results that approximate truth size, accuracy, precision reliability, reproducibility, appropriateness, bias statistical descriptions trade-offs, limiting factors, cost Grade components Quality (Validity) The quality of evidence indicates the extent to which one can be confident that an estimate of effect is correct. Strength (Benefit/Risk - Results) The strength of a recommendation indicates the extent to which one can be confident that adherence to the recommendation will do more good than harm. Implementation and application Will the results help me with my patient care? (Relevance & Prevalence)

6. Grading evidence process

7. Evaluating Diagnosis Studies:Validity (Quality) Are the results Valid? (grading quality) Was there an independent, blind comparison with a reference standard? (gold standard) Did the patient sample include an appropriate spectrum of the sort of patients to whom the diagnostic test will be applied in clinical practice? Is there a standard method for doing the test? (reproducibility, reliability) After framing a PICO question, performing a search, next step is critical appraisal: Were the results valid?Methods: reporting Were the methods for performing the test described in sufficient detail to permit replication of the test in your setting? REPRODUCIBILITY:  Was the test described clearly enough that you could perform it in your setting?  Were the analysis and interpretation of results clearly described?  Otherwise, you can't be sure of the same results.  Did the paper describe a standard method for doing the test? If so, you can make sure that whoever carries out the test for you follows the same method and thus should get similar results. If not, you can't rely on being able to duplicate the results. Blinding:  Was there an independent, blind comparison with a reference standard? E.g. the person interpreting the study test was blinded to results of the gold standard test, and vice versa.   How good was the method of blinding?  Lack of blinding may result in several forms of Review bias. Did the authors do a 'blind' comparison with a reference standard? The reference standard would be either an existing test known to be reliable, or else the presence or absence of a clinical condition being tested for. That is, the reference standard is reliable enough to define whether someone has the condition or not. If there is no reference standard (as sometimes happens), then that means the condition tested for is not well-defined, and it makes no sense to give numerical values for the reliability of the test.  The comparison with the reference standard, if it is done at all, should be done 'blind' - that is, those doing or interpreting the reference standard test should not know how their patients did on the test being compared with it. BLIND COMPARISON:  You want to compare the test being studied to the reference standard, the recognized preferred test for the condition being tested for.  You want to make sure that BOTH tests were applied independently to ALL patients.  Avoided expectation bias? By blinding pathologists to other test results and info Reference standard (gold standard) comparison:  The gold standard must be valid. All patients should be investigated by both tests. What are the results? As described in the Important Concepts section for diagnosis, the results you'd like to see are the positive and negative Likelihood Ratios.  However, it is far more likely that studies will report Sensitivity and Specificity.  From these, you can compute Likelihood Ratios.  You will also want to know how precise the results of the study using confidence intervals .Are the results of the test useful? A test should increase or decrease the probability of a disease's presence or absence to allow a clinical decision about therapy (or further more intensive testing) to be made. LR+/LR- Accounting:  Do the numbers add up?  The article should provide a clear description of patient flow through the diagnostic protocol and all patients should be accounted for Precision:  Confidence intervals on outcomes (sensitivity, specificity etc.) should be presented.  If not, calculate them. Bias Spectrum Bias:   Did the study sample include an appropriate spectrum of patients and was the population similar to yours?  If not, how would you expect the test to perform in your setting Verification Bias:   Did the results of the test being evaluated influence the decision to perform the gold standard test? For interested readers:  BENEFITS:  Are there risks associated with the test?  Are these outweighed by the danger of an undiagnosed disease? Diagnostic test performance is usually expressed as sensitivity, specificity and predictive values.  Make sure you understand these terms before moving on.  Likelihood ratios (LR) are the most valuable way to express a test's potential clinical impact.  The LR is a statistical expression. It describes how a test result effects the probability that a disease or condition is present.  Test information is only valuable if it changes the probability of disease enough to alter treatment or diagnosis.  Although the LR can be derived from sensitivity and specificity, those expressions alone don’t provide information on the effect of a result on disease probability. Calculate  LR's, if not provided in your article LR's are determined from a contingency table derived from the results, or more simply,  by the reported sensitivity and specificity.  Use an online calculator if you need  help using a contingency table. You can now determine the LR's. The LR for a positive result = sensitivity/ (1-specificity) The LR for a negative result = (1-sensitivity)/specificity Confidence Intervals should be provided for all of the above mentioned result measurements.   If not provided, provided calculate confidence intervals on the LR.  Will the results help me in patient care?In addition to the concerns raised earlier, you will want to be aware of the following issues.Were the patients being tested like my patients? Your patient should be similar in whatever way is likely to be relevant to this condition and its diagnosis (age, severity of condition, state of disease process, current treatment regime, etc.) to those in the study. PATIENTS:  Are your patients similar enough that the prevalence of the disease in the study population is similar to that in your patients?  Is the severity of the disease in the test population similar to patients you are likely to see? Consider the following Is your patient  typical of a study patient?  If not ,  the results may not be applicable. See discussion on spectrum bias above. Will the reproducibility of the test result and its interpretation be satisfactory in your setting?  If  the test was conducted or interpreted by individuals with specialized training, on different equipment, or in a different environment, can you expect similar results? Tests which require subjective interpretation may be problematic. Look for an analysis or discussion of inter-rater reliability (kappa), or perform a separate search on this.    Will the results change my management?  If the test result is positive will it change your diagnosis or  treatment? If its negative?  As few tests are perfect, the possibility that your test result is false (FP or FN) must be considered. Thus, most tests don’t rule in or rule out disease, they change the probability of disease. Consequently the real questions are 1) what is the probability of disease if the test result is positive (or negative)?  and 2) will a positive or negative result change the probability of disease enough to result in an important change in diagnosis or management. To answer these questions you must determine (or estimate) the pre-test probability of disease, and determine the likelihood ratio's for  positive and negative test results.  When you have determined these values you can determine the post-test probability of disease, then make a value judgment as to the potential benefits or risks of testing. Use the LR nomogram from CEBM. After framing a PICO question, performing a search, next step is critical appraisal: Were the results valid?

8. Judgments about Evidence Quality Since test accuracy may vary noticeably across patient populations, the panel needs to consider how well the patients population included in the studies corresponds to the patient population that is the subject of the recommendations and whether the diagnostic strategies were compared directly or indirectly to a common (reference) standard. Fletcher RH. Carcinoembryonic antigen. Ann Intern Med 1986;104(1):66-73. 27. Hlatky MA, Pryor DB, Harrell FE, Jr., Califf RM, Mark DB, Rosati RA. Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariable analysis. Am J Med 1984;77(1):64-71. 28. Levy D, Labib SB, Anderson KM, Christiansen JC, Kannel WB, Castelli WP. Determinants of sensitivity and specificity of electrocardiographic criteria for left ventricular hypertrophy. Circulation 1990;81(3):815-20. Since test accuracy may vary noticeably across patient populations, the panel needs to consider how well the patients population included in the studies corresponds to the patient population that is the subject of the recommendations and whether the diagnostic strategies were compared directly or indirectly to a common (reference) standard. Fletcher RH. Carcinoembryonic antigen. Ann Intern Med 1986;104(1):66-73. 27. Hlatky MA, Pryor DB, Harrell FE, Jr., Califf RM, Mark DB, Rosati RA. Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariable analysis. Am J Med 1984;77(1):64-71. 28. Levy D, Labib SB, Anderson KM, Christiansen JC, Kannel WB, Castelli WP. Determinants of sensitivity and specificity of electrocardiographic criteria for left ventricular hypertrophy. Circulation 1990;81(3):815-20.

9. Types of studies to evaluate a test or diagnostic strategy: outcome-based Example: RCT that explored a diagnostic strategy guided by the use of B-type natriuretic peptide (BNP) Designed to provide a more accurate diagnosis of heart failure - in patients presenting to the emergency department with acute dyspnea. The group randomized to receive BNP spent a shorter time in the hospital at lower cost, with no increased mortality or morbidity. When diagnostic intervention studies - ideally RCTs but also observational studies - comparing alternative diagnostic strategies with assessment of direct patient important outcomes are available, guideline recommendation panels can use the GRADE approach established for treatment questions. Figure 1 (legend). Two generic ways in which one can evaluate a test or diagnostic strategy: (a) patients are randomized to a new test or strategy or to an old test or strategy. Those with a positive test (cases detected) are randomized (or were previously randomized) to receive the best available treatment or no treatment. Investigators evaluate and compare patient important outcomes in all patients in both groups6; (b) patients receive a new test and a reference test (old test or strategy). Investigators can then calculate the accuracy of the test compared to the reference test. To make judgments about patient-importance of this information, patients with a positive test (or strategy) in either group are (or have been in previous studies) randomised to treatment or no treatment; Investigators evaluate and compare patient important outcomes in all patients in both groups. If randomized trials (RCTs) are not feasible or available, guideline panels must focus on studies of test accuracy and make inferences about the likely impact on patient-important outcomes. Figure 1 illustrates two generic study structures that investigators can use to evaluate the impact of testing. Mueller C, Scholer A, Laule-Kilian K, Martina B, Schindler C, Buser P, et al. Use of B-type natriuretic peptide in the evaluation and management of acute dyspnea. N Engl J Med 2004;350(7):647-54.Figure 1 (legend). Two generic ways in which one can evaluate a test or diagnostic strategy: (a) patients are randomized to a new test or strategy or to an old test or strategy. Those with a positive test (cases detected) are randomized (or were previously randomized) to receive the best available treatment or no treatment. Investigators evaluate and compare patient important outcomes in all patients in both groups6; (b) patients receive a new test and a reference test (old test or strategy). Investigators can then calculate the accuracy of the test compared to the reference test. To make judgments about patient-importance of this information, patients with a positive test (or strategy) in either group are (or have been in previous studies) randomised to treatment or no treatment; Investigators evaluate and compare patient important outcomes in all patients in both groups. If randomized trials (RCTs) are not feasible or available, guideline panels must focus on studies of test accuracy and make inferences about the likely impact on patient-important outcomes. Figure 1 illustrates two generic study structures that investigators can use to evaluate the impact of testing. Mueller C, Scholer A, Laule-Kilian K, Martina B, Schindler C, Buser P, et al. Use of B-type natriuretic peptide in the evaluation and management of acute dyspnea. N Engl J Med 2004;350(7):647-54.

10. Types of studies to evaluate a test or diagnostic strategy: accuracy-based Diagnostic accuracy is a surrogate outcome for what we are really interested in, which is patient important benefit and harm. Example: consistent evidence from well- designed studies of fewer false negative results with non-contrast helical CT than with IVP in the diagnosis of acute urolithiasis. However, the stones in the ureters “missed” by IVP are smaller, and hence are likely to pass more easily. Since randomized trials evaluating outcomes in patients treated for smaller stones are not available, the extent to which CT would reduce “missed” cases (false negatives) and have important health benefits - remains uncertain. Figure 1 (legend). Two generic ways in which one can evaluate a test or diagnostic strategy: (a) patients are randomized to a new test or strategy or to an old test or strategy. Those with a positive test (cases detected) are randomized (or were previously randomized) to receive the best available treatment or no treatment. Investigators evaluate and compare patient important outcomes in all patients in both groups6; (b) patients receive a new test and a reference test (old test or strategy). Investigators can then calculate the accuracy of the test compared to the reference test. To make judgments about patient-importance of this information, patients with a positive test (or strategy) in either group are (or have been in previous studies) randomised to treatment or no treatment; Investigators evaluate and compare patient important outcomes in all patients in both groups. If randomized trials (RCTs) are not feasible or available, guideline panels must focus on studies of test accuracy and make inferences about the likely impact on patient-important outcomes. Figure 1 illustrates two generic study structures that investigators can use to evaluate the impact of testing. Deeks JJ. Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests. Bmj 2001;323(7305):157-62. Worster A, Preyra I, Weaver B, Haines T. The accuracy of noncontrast helical computed tomography versus intravenous pyelography in the diagnosis of suspected acute urolithiasis: a meta-analysis. Ann Emerg Med 2002;40(3):280-6. Worster A, Haines T. Does replacing intravenous pyelography with noncontrast helical computed tomography benefit patients with suspected acute urolithiasis? Can Assoc Radiol J 2002;53(3):144-8.Figure 1 (legend). Two generic ways in which one can evaluate a test or diagnostic strategy: (a) patients are randomized to a new test or strategy or to an old test or strategy. Those with a positive test (cases detected) are randomized (or were previously randomized) to receive the best available treatment or no treatment. Investigators evaluate and compare patient important outcomes in all patients in both groups6; (b) patients receive a new test and a reference test (old test or strategy). Investigators can then calculate the accuracy of the test compared to the reference test. To make judgments about patient-importance of this information, patients with a positive test (or strategy) in either group are (or have been in previous studies) randomised to treatment or no treatment; Investigators evaluate and compare patient important outcomes in all patients in both groups. If randomized trials (RCTs) are not feasible or available, guideline panels must focus on studies of test accuracy and make inferences about the likely impact on patient-important outcomes. Figure 1 illustrates two generic study structures that investigators can use to evaluate the impact of testing. Deeks JJ. Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests. Bmj 2001;323(7305):157-62. Worster A, Preyra I, Weaver B, Haines T. The accuracy of noncontrast helical computed tomography versus intravenous pyelography in the diagnosis of suspected acute urolithiasis: a meta-analysis. Ann Emerg Med 2002;40(3):280-6. Worster A, Haines T. Does replacing intravenous pyelography with noncontrast helical computed tomography benefit patients with suspected acute urolithiasis? Can Assoc Radiol J 2002;53(3):144-8.

11. Judgments about Evidence Quality very strong association, up 2 levels insulin in diabetic ketoacidosis strong, consistent association with no plausible confounders, up 2 levels fluoride for preventing cavities strong association can move up 1 level ? HRT ? very strong association, up 2 levels insulin in diabetic ketoacidosis strong, consistent association with no plausible confounders, up 2 levels fluoride for preventing cavities strong association can move up 1 level ? HRT ?

12. Judgments about Evidence Quality We have noted that RCTs involving direct comparison of the impact of alternative diagnostic strategies on patient important outcomes, such as the randomized trial of BNP in heart failure provide, in the absence of limitations in design and conduct, imprecision, inconsistency, indirectness and reporting bias, yield high quality evidence. Guyatt G, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is 'quality of evidence' and why is it important to clinicians? BMJ submitted. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454):1490. Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, et al. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 2006;174(5):605-14. Evidence is likely to be lower quality if only studies of diagnostic accuracy are available. While valid accuracy studies start as high quality in the diagnostic framework, such studies are extremely vulnerable to limitations as a result of the indirect evidence they may provide regarding impact on patient-important outcomes. We have noted that RCTs involving direct comparison of the impact of alternative diagnostic strategies on patient important outcomes, such as the randomized trial of BNP in heart failure provide, in the absence of limitations in design and conduct, imprecision, inconsistency, indirectness and reporting bias, yield high quality evidence. Guyatt G, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is 'quality of evidence' and why is it important to clinicians? BMJ submitted. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454):1490. Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, et al. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 2006;174(5):605-14. Evidence is likely to be lower quality if only studies of diagnostic accuracy are available. While valid accuracy studies start as high quality in the diagnostic framework, such studies are extremely vulnerable to limitations as a result of the indirect evidence they may provide regarding impact on patient-important outcomes.

13. Coronary CT scanning versus invasive angiography: arriving at a bottom line for study quality Table 4 shows the evidence profile and the quality assessment for all outcomes of CT angiography in comparison to invasive angiography. Insert Table 4 approximately here The original accuracy studies were well planned and executed, and we are confident that limitations in test accuracy will have deleterious consequences on patient important outcomes (that is, we have no concerns about directness), the results are precise, and we do not suspect important reporting bias. There are, however, problems with inconsistency. Reviewers addressing the relative merits of CT versus invasive angiography for diagnosis of coronary disease found important heterogeneity in the results for the proportion of target-negative patients with a positive test result (specificity) that they could not explain (Figure 3). This heterogeneity reduces our confidence in estimate of true and false positives, and leads to downgrading from high to moderate quality evidence. Reviewers addressing the relative merits of CT versus invasive angiography for diagnosis of coronary disease found important heterogeneity in the results for the proportion of target-negative patients with a positive test result (specificity) that they could not explain (Figure 3). This heterogeneity reduces our confidence in estimate of true and false positives, and leads to downgrading from high to moderate quality evidence Hamon M, Biondi-Zoccai GG, Malagutti P, Agostoni P, Morello R, Valgimigli M, et al. Diagnostic performance of multislice spiral computed tomography of coronary arteries as compared with conventional invasive coronary angiography: a meta-analysis. J Am Coll Cardiol 2006;48(9):1896-910. Table 4 shows the evidence profile and the quality assessment for all outcomes of CT angiography in comparison to invasive angiography. Insert Table 4 approximately here The original accuracy studies were well planned and executed, and we are confident that limitations in test accuracy will have deleterious consequences on patient important outcomes (that is, we have no concerns about directness), the results are precise, and we do not suspect important reporting bias.

14. Evaluating Diagnosis Studies:Strength (Results) What are the Results? (grading strength) What is the magnitude of benefit, and how reliable/precise are these results? What are the magnitudes of risk, burden, and cost; and how reliable/precise are these results? Do the benefits outweigh the risks/burdens/costs? Are there known trade-offs? Are there unknown possible trade-offs? Result Parameters: Accuracy, Likelihood Ratios, Confidence Intervals Are likelihood ratios for the test results presented or data necessary for their calculation included? Statistics for Pre-test Probability (prevalence), Likelihood Ratios, Sensitivity and Specificity, Predictive Values Are the results of the test useful? Factors that influence strength: Evidence for less serious event than one hopes to prevent Smaller Treatment Effect Imprecise Estimate of Treatment Effect Low Risk of Target Event Higher Risk of Therapy Higher Costs Varying Values Higher Burden of Therapy Parameters: -Prevalence -Pre-Test Probability -Sensitivity & Specificity -Accuracy -Positive Predictive Value and Negative Predictive Value -Likelihood Ratios (LR+, LR-) -Likelihood Ratio Nomogram -Post-Test Probability -Test Result Thresholds After framing a PICO question, performing a search, next step is critical appraisal: Were the results valid?Methods: reporting Were the methods for performing the test described in sufficient detail to permit replication of the test in your setting? REPRODUCIBILITY:  Was the test described clearly enough that you could perform it in your setting?  Were the analysis and interpretation of results clearly described?  Otherwise, you can't be sure of the same results.  Did the paper describe a standard method for doing the test? If so, you can make sure that whoever carries out the test for you follows the same method and thus should get similar results. If not, you can't rely on being able to duplicate the results. Blinding:  Was there an independent, blind comparison with a reference standard? E.g. the person interpreting the study test was blinded to results of the gold standard test, and vice versa.   How good was the method of blinding?  Lack of blinding may result in several forms of Review bias. Did the authors do a 'blind' comparison with a reference standard? The reference standard would be either an existing test known to be reliable, or else the presence or absence of a clinical condition being tested for. That is, the reference standard is reliable enough to define whether someone has the condition or not. If there is no reference standard (as sometimes happens), then that means the condition tested for is not well-defined, and it makes no sense to give numerical values for the reliability of the test.  The comparison with the reference standard, if it is done at all, should be done 'blind' - that is, those doing or interpreting the reference standard test should not know how their patients did on the test being compared with it. BLIND COMPARISON:  You want to compare the test being studied to the reference standard, the recognized preferred test for the condition being tested for.  You want to make sure that BOTH tests were applied independently to ALL patients.  Avoided expectation bias? By blinding pathologists to other test results and info Reference standard (gold standard) comparison:  The gold standard must be valid. All patients should be investigated by both tests. What are the results? As described in the Important Concepts section for diagnosis, the results you'd like to see are the positive and negative Likelihood Ratios.  However, it is far more likely that studies will report Sensitivity and Specificity.  From these, you can compute Likelihood Ratios.  You will also want to know how precise the results of the study using confidence intervals .Are the results of the test useful? A test should increase or decrease the probability of a disease's presence or absence to allow a clinical decision about therapy (or further more intensive testing) to be made. LR+/LR- Accounting:  Do the numbers add up?  The article should provide a clear description of patient flow through the diagnostic protocol and all patients should be accounted for Precision:  Confidence intervals on outcomes (sensitivity, specificity etc.) should be presented.  If not, calculate them. Bias Spectrum Bias:   Did the study sample include an appropriate spectrum of patients and was the population similar to yours?  If not, how would you expect the test to perform in your setting Verification Bias:   Did the results of the test being evaluated influence the decision to perform the gold standard test? For interested readers:  BENEFITS:  Are there risks associated with the test?  Are these outweighed by the danger of an undiagnosed disease? Diagnostic test performance is usually expressed as sensitivity, specificity and predictive values.  Make sure you understand these terms before moving on.  Likelihood ratios (LR) are the most valuable way to express a test's potential clinical impact.  The LR is a statistical expression. It describes how a test result effects the probability that a disease or condition is present.  Test information is only valuable if it changes the probability of disease enough to alter treatment or diagnosis.  Although the LR can be derived from sensitivity and specificity, those expressions alone don’t provide information on the effect of a result on disease probability. Calculate  LR's, if not provided in your article LR's are determined from a contingency table derived from the results, or more simply,  by the reported sensitivity and specificity.  Use an online calculator if you need  help using a contingency table. You can now determine the LR's. The LR for a positive result = sensitivity/ (1-specificity) The LR for a negative result = (1-sensitivity)/specificity Confidence Intervals should be provided for all of the above mentioned result measurements.   If not provided, provided calculate confidence intervals on the LR.  Will the results help me in patient care?In addition to the concerns raised earlier, you will want to be aware of the following issues.Were the patients being tested like my patients? Your patient should be similar in whatever way is likely to be relevant to this condition and its diagnosis (age, severity of condition, state of disease process, current treatment regime, etc.) to those in the study. PATIENTS:  Are your patients similar enough that the prevalence of the disease in the study population is similar to that in your patients?  Is the severity of the disease in the test population similar to patients you are likely to see? Consider the following Is your patient  typical of a study patient?  If not ,  the results may not be applicable. See discussion on spectrum bias above. Will the reproducibility of the test result and its interpretation be satisfactory in your setting?  If  the test was conducted or interpreted by individuals with specialized training, on different equipment, or in a different environment, can you expect similar results? Tests which require subjective interpretation may be problematic. Look for an analysis or discussion of inter-rater reliability (kappa), or perform a separate search on this.    Will the results change my management?  If the test result is positive will it change your diagnosis or  treatment? If its negative?  As few tests are perfect, the possibility that your test result is false (FP or FN) must be considered. Thus, most tests don’t rule in or rule out disease, they change the probability of disease. Consequently the real questions are 1) what is the probability of disease if the test result is positive (or negative)?  and 2) will a positive or negative result change the probability of disease enough to result in an important change in diagnosis or management. To answer these questions you must determine (or estimate) the pre-test probability of disease, and determine the likelihood ratio's for  positive and negative test results.  When you have determined these values you can determine the post-test probability of disease, then make a value judgment as to the potential benefits or risks of testing. Use the LR nomogram from CEBM. Factors that influence strength: Evidence for less serious event than one hopes to prevent Smaller Treatment Effect Imprecise Estimate of Treatment Effect Low Risk of Target Event Higher Risk of Therapy Higher Costs Varying Values Higher Burden of Therapy Parameters: -Prevalence -Pre-Test Probability -Sensitivity & Specificity -Accuracy -Positive Predictive Value and Negative Predictive Value -Likelihood Ratios (LR+, LR-) -Likelihood Ratio Nomogram -Post-Test Probability -Test Result Thresholds After framing a PICO question, performing a search, next step is critical appraisal: Were the results valid?

15. Benefit/risk/harm/cost balanceStrength Net benefits: The test intervention does more good than harm. Trade-offs: There are important trade-offs between the benefits and harms. Uncertain trade-offs: Not clear whether the test intervention does more good than harm. No net benefits: The test intervention does not do more good than harm. Net harms: The test intervention does more harm than good. Judgements about the strength of a recommendation require consideration of the balance between benefits and harms, the quality of the evidence, translation of the evidence into specific circumstances, and the certainty of the baseline risk. It is also important to consider costs (resource utilisation) prior to making a recommendation.We suggest using evidence profiles as the basis for making the judgements outlined above   Publications List of GRADE working group publications: Manuscripts 1. Schünemann HJ, Best D, Vist G, Oxman AD for the GRADE working group. Letters, numbers, symbols, and words: How best to communicate grades of evidence and recommendations? Canadian Medical Association Journal (CMAJ) 2003;169(7):677-680 For a full text version of this article, please go to the CMAJ website http://www.cmaj.ca/cgi/reprint/169/7/677  or download a PDF version from this link: http://www.cmaj.ca/cgi/reprint/169/7/677.pdf and for online tables related to the article from:http://www.cmaj.ca/cgi/data/169/7/677/DC1/1 2. The GRADE* Working Group. Grading quality of evidence and strength of recommendations. BMJ 2004;328:1490-1494 (printed, abridged version). For the non-abridged version of this article, please go to the BMJ website http://bmj.bmjjournals.com/cgi/content/full/328/7454/1490 or download a full PDF version from this link: http://bmj.bmjjournals.com/cgi/reprint/328/7454/1490.pdf  or, if you don't have access, here [pdf] 3. The GRADE* Working Group. Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches. BMC Health Serv Res. 2004 Dec 22;4(1):38. For the full text version, please follow this link http://www.biomedcentral.com/1472-6963/4/38 or download the full PDF version here: http://www.biomedcentral.com/content/pdf/1472-6963-4-38.pdf 4. The GRADE* Working Group. Systems for grading the quality of evidence and the strength of recommendations II: A pilot study of a new system for grading the quality of evidence and the strength of recommendations. BMC Health Serv Res. 2005 Mar 23;5(1):25. For the full text version, please follow this link http://www.biomedcentral.com/1472-6963/5/25 or download the full PDF version here:  http://www.biomedcentral.com/content/pdf/1472-6963-5-25.pdf   Abstracts and conferences 1. Oxman AD, Briss P, Eccles M, Grimshaw JM, Guyatt G, Phillips B, Schunemann H, Vist G, Zaza S. Grading evidence and recommendations. BMC Meeting Abstracts: 9th International Cochrane Colloquium 2001, 1:pc148   Articles, such as editorials and letters to the editor and relevant articles citing the work of the GRADE group 1. Commentary: Ross E.G. Upshur Are all evidence-based practices alike? Problems in the ranking of evidence. CMAJ, Sep 2003; 169: 672 - 673.http://www.cmaj.ca/cgi/content/full/169/7/672 (related to manuscript 1.) 2.http://www.cmaj.ca/cgi/content/full/169/7/649 (related to manuscript 1.) 3.http://www.cmaj.ca/cgi/eletters/169/7/677#457 4. P. Glasziou, J. Vandenbroucke, and I. Chalmers. Assessing the quality of research. BMJ, January 3, 2004; 328(7430): 39 - 41.http://bmj.bmjjournals.com/cgi/content/full/328/7430/39 (cites manuscript 1.)   Selected web sites citing the work of the GRADE group Endocrine Society Clinical Guidelines Underway (February 23, 2005) http://www.endo-society.org/publicpolicy/insider/clinical-guidelines-update.cfm

16. Decision Thresholds and Diagnostic Tests: Alternative Conceptualization Tests or test strategies that result in patients moving below the test threshold or above the treatment threshold (given the treatment exists) will often lead to strong recommendations despite the false negative and false positive test results. On the other hand, test or strategies that will only marginally change the probability of disease and require further testing will usually lead to weak recommendations. The test properties that best coincide with results that move the patient’s probability significantly up or down are the Likelihood Ratios. Figure 2 legend: What clinicians expect of a good test is that results change the probability sufficient to confirm or exclude a diagnosis. If a test result moves the probability below a threshold at which the disease is very unlikely and downsides associated with the treatment outweigh any anticipated benefit, then no further testing and no treatment are indicated (test threshold). If the test result increases the probability of disease above a threshold at which one would cease testing and begin treatment the clinician has crossed a test threshold. If the pre-test probability is above the treatment threshold, further confirmatory testing that raises the probability further would not be helpful. If the pre-test probability is below the test threshold, further exclusionary testing would not be useful. When the probability is between the test and treatment thresholds testing will be useful. Test results are of greatest value when they shift the probability across either threshold. Figure 2 legend: What clinicians expect of a good test is that results change the probability sufficient to confirm or exclude a diagnosis. If a test result moves the probability below a threshold at which the disease is very unlikely and downsides associated with the treatment outweigh any anticipated benefit, then no further testing and no treatment are indicated (test threshold). If the test result increases the probability of disease above a threshold at which one would cease testing and begin treatment the clinician has crossed a test threshold. If the pre-test probability is above the treatment threshold, further confirmatory testing that raises the probability further would not be helpful. If the pre-test probability is below the test threshold, further exclusionary testing would not be useful. When the probability is between the test and treatment thresholds testing will be useful. Test results are of greatest value when they shift the probability across either threshold.

17.    Judgments about OverallGrade of Recommendations

18. Conclusions The GRADE approach to grading the quality of evidence and strength of recommendations for diagnostic guidelines provides comprehensive and transparent methodology for developing these recommendations. The GRADE Working Group presents an overview of the approach, already established for grading treatment recommendations. Publication for Diagnostic GRADE in BMJ is pending. Extensive application to diagnostic guidelines is likely to refine the approach. The basic methodology & considerations that follow from recognizing test results as surrogate markers are unlikely to change.

19. Citations Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454):1490. Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, et al. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. Am J Respir Crit Care Med 2006;174(5):605-14. Schunemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Williams J, et al. GRADEing the quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ submitted. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann Intern Med 2003;138(1):40-4. http://www.stard-statement.org/website%20stard/ Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 2003;138(1):W1-12. http://www.stard-statement.org/website%20stard/ Bossuyt PM, Irwig L, Craig J, Glasziou P. Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ 2006;332(7549):1089-92. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 1999;282(11):1061-6. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 2006;174(4):469-76. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003;3:25. http://www.biomedcentral.com/1471-2288/3/25 Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, Kleijnen J. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol 2006;6:9. http://www.biomedcentral.com/1471-2288/6/9