Retrieving Actions in Group Contexts

1. Retrieving Actions in Group Contexts TianLan, Yang Wang, Greg Mori, Stephen Robinovitch Simon Fraser University Sept. 11, 2010

2. Outline • Action Retrieval as Ranking • Contextual Representation of Actions • Results and Future Work

4. Nursing Home • Fall analysis in nursing home surveillance videos • a system automatically rank the videos according to the relevance to fall action is expected

5. Action-Action Context What other people are doing ? Context

6. Actions in Group Context • Motivation • human actions are rarely performed in isolation, the actions of individuals in a group can serve as context for each other. • Goal • explore the benefit of contextual information in action retrieval in challenging real-world applications

7. Action Context Descriptor τ τ z + action Focal person Context action

8. Action Context Descriptor Feature Descriptor Multi-class SVM e.g. HOG by Dalal & Triggs score score score score max action class action class action class action class …

9. Outline • Action Retrieval as Ranking • Contextual Representation of Actions • Results and Future Work

10. Classification or Retrieval • Previous Work • Most work in human action understanding focuses on action classification.

11. Classification or Retrieval • Most surveillance tasks are typical retrieval tasks • retrieve a small video segment contains a particular action from thousands of hours of videos. • The “action of interest” is rare event • Extremely imbalanced classes

12. Query : fall Action Retrieval Rank according to the relevance to falls

13. Learning • Input: document-rank pair (xi,yi) • Optimization Joachims, KDD 06

14. Ranking SVM • Ranking function h(x) h(x) Rank r1 Rank r2 Rank r3

15. Action Retrieval - training irrelevant relevant very relevant

16. Outline • Action Retrieval as Ranking • Contextual Representation of Actions • Results and Future Work

17. Dataset • Nursing Home Dataset • 5 action categories: walking, standing, sitting, bending and falling. (per person) • 18 video clips. • Query: fall • Collective Activity Dataset (Choi et al. VS. 09) • 5 action categories: crossing, waiting, queuing, walking, talking. (per person) • 44 video clips. • Query: each of the five actions

18. Dataset • Nursing Home Dataset

19. Dataset • Collective Activity Dataset

20. System Overview u Person Detector Rank SVM Person Descriptor Video v • Pedestrian Detection • by Felzenszwalb et al. • Background Subtraction • HOG by Dalal & Triggs • LST by Loy et al. • at cvpr 09

21. Baselines • Context vs No Context • Action Context Descriptor • Original feature descriptors, e.g. HOG (Dalal & Triggs at CVPR 05), LST (Loy et al. at CVPR 09) • RankSVMvs SVM • Methods • Context + RankSVM (our method) • Context + SVM • No Context + RankSVM • No Context + SVM

23. Retrieval Results Nursing Home Dataset

24. Retrieval Results Collective Activity Dataset

25. Retrieval Results Collective Activity Dataset

26. Retrieval Results Collective Activity Dataset

27. 1 2 3 4

28. 5 6 7 8

29. Action Classification [10] Choi et al. in VS. 09 Collective Activity Dataset

30. Conclusion • A new contextual feature descriptor to represent actions • action context (AC) descriptor • Formulate our problem as a retrieval task.

31. Future Work • Contextual Feature Descriptors • How to only encode useful context? • Rank-SVM loss, optimize the NDCG score

32. Thank you!

34. 5 6 7 8