The index test results: positivity and negativity criteria. The cut-off value

1. The index test results: positivity and negativity criteria. The cut-off value Agostino Colli Gargnano April 4-8 2017

2. INDEX TEST : positive or negative (a binary classification ) Diagnostic tests’ results are expressed as : • a continuous variable (a measure) 2. a description (radiology,endoscopy, histology, microbiology etc)

3. Dicotomizing a continuous variable + Disease: severe disease moderate mild unsure unsure other disease healthy - Disease POSITIVE NEGATIVE To dichotomize a continuous variable is a compromise, some information are unavoidably lost

4. Dicotomizing a continuous variable not affected affected • there is always some overlap between values in affected and non affected • there are always some FN and some FP i.e. sensitivity and specificity are always < 100% • changing the cut off value you can improve sensitivity at the expense of specificity and specificity at the expense of sensitivity i.e. reducing FN , FP increase.

5. no FN the index test is POSITIVE when < 909 fewFP many FN many FP

6. For diagnostic test producing dichotomous results , the positivity/negativity criteria must be defined. Lobar consolidation with pleural effusion . What criteria are needed for the diagnosis ? If all signs are needed few FP (many FN) If only one sign is needed few FN (many FP) Focal liver lesion with arterial contrast enhancement. What criteria (venous wash out, cirrhosis) are needed for the diagnosis?

7. At least 3 points : sensitivity 91.4% specificity 98.7% N Engl J Med 2017;376:957-70. LR+ = 91.4/1.3 = 70 LR- = 8.6/98.7= 0.09

8. Choose “ THE BEST CUT OFF VALUE “! • ..but what is “the best cut off “? • The cut off value associated with: • the best accuracy i.e. with less false results FN and FP ( Youden index ) • the lowest LR- (= very few FN) • the highest LR+ ( = very few FP) • For 1 FP and FN have the same relevance; for 2 it is optimal to have the lowest number of FN ; for 3 the lowest number of FP. • The question is : has the same consequences to be FP or FN ? Usually the consequences are not the same. • Youden index = sensitivity+specificity-1 = TPR – FPR • The highest value = the best cut off value

9. Youden Index 73 78 71 LR- 0.06 0.10 0.25 If to be FP or FN is the same, choose 11.7 kPa ( the highest Youden index) To reduce the number of FN, choose 9.4 kPa To reduce the number of FP, choose 17.1 kPa reducing FN: ruling OUT the diagnosis; reducing FP: ruling IN the diagnosis

10. the penalty for being wrong It is better (or less worse) : to treat false positive patients to withhold treatment in false negative

11. The “optimal cut off value”, another definition Semin Nucl Med 1978; 8: 283 the value that maximizes the overall expected benefit the point on the ROC curve at which the slope of the curve is equal to: where H is harm, B benefit and p(D) the prevalence of the target condition “Benefit” was defined as the net advantage of being treated in the presence of the target disease (i.e. a true positive case), while “harm” was defined as any damage due to treatment in the absence of the target disease (i.e. a false positive). from maximal accuracy to maximal benefit

12. H/B = 1/3.7 cut off value = 9.4 kPa H/B = 1/10 cut off value= 6.8 kPa Colli A. PLoS ONE 11(10): e0164452. doi:10.1371/journal.pone.0164452

13. RISK OF BIAS In a diagnostic study the cut off value can be derived as the “best cut off from the results in the included patients . In this case the accuracy will be overestimated. (high risk of bias ). On the contrary if a diagnostic study is designed to validate a predefined cut off value of the index test, the accuracy estimates will be not biased. ( low risk of bias ). In the case of derivation the cut off value is a posteriorI chosen as the best and fits perfectly with the results In the case of valdation The setting is different such as patients and their results and the predefind value cannot perfectly fit. See QUADAS 2

14. The index test results can be expressed as dichotomous i.e. pos/neg or as a continuous variable which have to be dichotomized. • For imaging , histological, endoscopic test the positivity /negativity criteria have to be defined. • Score systems use a semi-quantitative approach with a defined cut off value • Dichotomizing a continuous variable, some information are lost • Due to an overlap between diseased and non diseased and a trade off between sensitivity and specificity changing a cut of value to improve sensibility impairs specificity and vice versa. • The best cut off value can be the one that ensure the best accuracy and can be derived according to Youden index • If it is relevant to reduce false negative results ,the best cut off is the one with the lowest LR- • If it is relevant to reduce false positive results ,the best cut off is the one with the highest LR+ • Taking into account the clinical consequences of testing the best cut off can be defined as the one that ensures the maximal expected benefit • Studies using cut off value of the index test derived from the included patients are at risk of bias i.e. of overestimating the accuracy. Studies which validated a predefined cut off value are at low risk of bias.