Post-Hoc Scoring of Topic 102

1. Post-Hoc Scoring of Topic 102

2. Topic 102 • Documents referring to marketing or advertising restrictions proposed for inclusion in, or actually included in, the Master Settlement Agreement (“MSA"), including, but not limited to, restrictions on advertising on billboards, stadiums, arenas, shopping malls, buses, taxis, or any other outdoor advertising.

3. Topic 102 Sampling

4. Judged (Adjudicated) Samples Exist for 8 of the 107 Submitted Documents Precision ≈ 7/8 = 0.875

5. Estimating Recall • Estimated total relevant: 562,402 • Plus or minus 100,000! (at 95% confidence) • 385,000 of these are from Stratum 8 (NNN)! • Estimate 68.5 documents found • 7 assessed documents, each scaled by 1/P • This is practical because none were in stratum 8 • Recall = 68.5/562402 = 0.000122 ≈ 0 • Ignoring stratum 8, Recall = 68.5/176990 = 0.00039

6. Takeaway Messages • You can get high precision manually • But achieving high recall requires automation • You can measure precision easily • But measuring recall requires sampling • Post-hoc sampling computes relative recall • Sampling gets too sparse in very large collections • Assessor errors are greatly magnified • Systems will rarely hit on an assessed document

7. Structured Evaluation Exercise • Two systems • Boolean Queries • Interactive result set • Draw 200 samples from the four strata • Undersample the NN stratum • Compute Precision and Recall as above