Enhancing Text Categorization with Unobserved Labels and Structured Aspect Variables

1. Aspect Guided Text Categorization with Unobserved Labels Dan Roth, YuanchengTu University of Illinois at Urbana-Champaign

2. Text Categorization Sports Health Business … Science C1 C2 C3 … CN • An archetypical Multi-Class Classification (MCC) problem • F : X → Y • a document, d ∈ X , a collection of classes Y = {c1, c2, . . . , cN}

3. Motivation: what are we missing? C1 C2 C3 … CN Sports Health Business … Science Class labels (Y) contain information which can help classification How can we explore the label space?

4. Car Navigation Command Classification X Y

5. Aspect Variables Show me where I can eat nearby X Find nearest restaurant findnearestrestaurant Null Null Y Manner z5 z3 z4 z1 z2 Action Detail Modifier Topic

6. Significance of the Aspect Variables If Topic = “restaurant”, then Action ≠ “turn” Observed Label 1. turn on the radio 2. GPS navigation Unobserved Label turn on GPS • Predicting better aspects implies predicting better class labels • Adding constraints to the aspect space • Predicting previously unobserved labels

7. Outline • Car Command Text Categorization Task • Data and aspects • Unobserved labels • Constrained Conditional Model (CCM) • Aspects variables to introduce constraints • Objective function • Training and Inference • Experimental Results • Standard multiclass classification setting • Predicting Unobserved Labels • Conclusion

8. Adding Constraints by Hidden Aspects x3 x4 x1 x5 x2 x7 x6 y1 Y Intuition: introduce structure on hidden variables X

9. Adding Constraints Through Hidden Aspects Z2 Z1 Z3 Z4 x3 x4 x1 x5 x2 x7 x6 y1 Y Z5 Use constraints to capture the dependencies X

10. Penalty for violating the constraint. Weight Vector for “local” learners How far away is y from a “legal” assignment Aspect functions Objective Function of CCM

11. Training and Inference • Learning + Inference (L+I) • Ignore constraints during training • Inference Based Training (IBT) • Consider constraints during training • References to CCM (aka ILP formulation) • Roth&Yih04, Has been shown useful in the context of many NLP problems: SRL, Summarization; Co-reference; Information Extraction; Transliteration • 07; Punyakanok et.al 05,08; Chang et.al 07,08; Clarke&Lapata06,07; Denise&Baldrige07;Goldwasser&Roth'08; Martin,Smith&Xing'09

12. Z2 Z3 Z1 Z4 Z5 x3 x4 x1 x5 x2 x7 x6 Learning + Inference Learning + Inference (L+I) Learn models independently f1(x) f2(x) f3(x) Y f4(x) f5(x) X

13. True Global Labeling Y -1 1 -1 -1 1 Local Predictions Apply Constraints: Y’ -1 1 -1 1 1 x3 x4 x1 x5 x2 x7 x6 Y’ -1 1 1 1 1 Inference Based Training Example: Perceptron-based Global Learning f1(x) X f2(x) f3(x) Y f4(x) f5(x)

14. Outline • Car Command Text Categorization Task • Data and aspects • Unobserved labels • Constrained Conditional Model (CCM) • Aspects variables to introduce constraints • Objective function • Training and Inference • Experimental Results • Standard multiclass classification setting • Predicting Unobserved Labels • Conclusion

15. Car Navigation Command Classification X Y

16. Evaluation Metrics • Standard Accuracy • The percentage of correctly labeled examples • Weighted Aspect-based Metric (WAM) • A weighted Hamming distance computed at the aspect level

17. Experiments and Evaluation • Standard MCC Setting

18. Experiments and Evaluation • Standard MCC Setting

19. Experiments and Evaluation • Predicting Unobserved Labels Observed Label 1. turn on the radio 2. GPS navigation Unobserved Label turn on GPS

20. Conclusion • Summary • Text Categorization with a meaningful, structured label space • A model that exploits the structure by adding hidden aspect variables • Adding constraints and reformulating the task as a structure prediction problem • Predicting unobserved new labels

21. Thank You! AND Questions?