4 .4 Object Recognition in Videos

1. Model-based Object Recognition • Object Segmentation • Classification of Objects • Analysis of the Curvature of a Shape • Curvature Scale Space Features • Experimental Results 4.4 Object Recognition in Videos

2. Characteristic features (e.g., histogram, shape) Object model A) Model object C) Compare B) Extract features Model-based Object Recognition Unknown image Object

3. * Scaling * Rotation * Noise * Blurring * Perspective transformation * Deformation Goal and Challenges • Goal: Recognition of objects in videos • Major challenges: camera motion, variation of objects

4. Object Segmentation (1) • Assumption: At least half of the visible area in each frame is background. • Estimate the camera motion between consecutive frames. • Calculate the parameters of the camera model:

5. Object Segmentation (2) • Assumption: At least half of the visible area in each frame is background. • Estimate the camera motion between consecutive frames. • Calculate parameters of the camera model:

6. Backgroundimage • Apply a median filter on the transformed frames to • construct the background image. • Compare the background image with the transformed frame to get the object mask. Segmentation object mask (object shape) Object Segmentation (3) frames • Calculate parameters of the camera model: Camera Motion

7. Identify Shape Features Segmentation Parameterizationof the shape Calculate the curvature of the shape

8. curve r r Definition of the Curvature (1) • The curvature of a curve at a given point has a mag-nitude equal to the reciprocal of the radius of an os-culating circle (the circle touching the curve): • The curvature is a vector pointing to the direction of the circle’s center. • A small circle corresponds with a high curvature’s magnitude, a straight line has a curvature of zero.

9. Definition of the Curvature (2) • Consider a plane curve u(t) that lies completely within a 2D plane. u(t) is parameter-ized by the arc length t. • The curve u(t) is parametrically defined by two functions x(t) and y(t): u(t) = (x(t), y(t)). • We define the curvature K for a plane curve u(t): • and define the first derivative (gradient) and the second derivative (change of the gradient) .

10. Definition of the Curvature (3) • We can now derive a less general definition of the curvature if we explicitly use plane curves defined by y = f(x). We get the following definition of the curvature for each point (x, f(x)): • This form is widely used in engineering, for example • to approximate the fluid flow around surfaces, e.g. in aerodynamics (gases) or hydrodynamics (liquids), • to derive the characteristic behavior when bending structural elements, e.g., put weight on a beam and analyze the flexure.

11. Definition of the Curvature (4) • Example • Consider the parameterized curve u(t) = (x(t), y(t)) = (t, t2). The explicit definition of this curve is: y = f(x) = x2. • Calculate curvature based on the parameterized curve:First and second derivatives: = 1, = 0, = 2t, = 2. • Alternative approach: Calculate curvature based on the explicit definition:

12. Definition of the Curvature (5) • Approximation of derivatives for discrete values (parameterized shapes): • Parameter t is defined for integer values • hx and hy normalize the derivatives depending on the distance between sample points.

13. Definition of the Curvature (6) Example of the curvature of a shape curvature position of the shape Problem: It is very difficult to match the curvature functions of two shapes. Idea:  Identify significant curvature features. We use the curvature scale space technique.

14. Curvature Scale Space: Smoothing in Iterations • Analyze the outer shape of an object. • Smooth the shape with a Gaussian kernel in a sequence of iterations. • The inflection points in each iteration are used as features to describe the object.

15. Curvature Scale Space Diagram • A curvature scale space diagram is a visual representation of the inflection points observed during the smoothing process. iterations 100 first shape pixel shape after 100 iterations arc length • The peaks are used as features to describe the object.

16. Properties of the Curvature Scale Space • Pro: * Only a few values are required to describe a complex object. * The approach is invariant to rotation or scaling. * Low computation time. • Contra: • * Bad classification results with some shapes.

17. Solution: Use position, height and width of each peak as a feature. Ambiguities of Curvature Scale Space Images (1) • Shallow vs. deep concavities: iterations iterations Figure 1 Figure 2 Scale Space Image 1 Scale Space Image 2

18. Ambiguities of Curvature Scale Space Images (2) • Poor representation of convex re-gions of a shape: convex objects are not represented at all. • Solution: Mapped shapes iterations arc length position offirst shape pixel

19. Mapped Shapes (1) • Idea: mirror each contour pixel at a circle around the object P (x,y) C

20. Mapped Shapes (2) • Strong convex segments of the original shape become concave segments of the mapped shape. (x‘,y‘) P (x,y) C

21. Standard Curvature Scale Space Diagram • Calculate standard curvature scale space features.

22. Add the Curvature Scale for the Mapped Shapes • Calculate features for the mapped shape.

23. Aggregation of the Classification Results • Similar objects are grouped in one object class. • Distance between input ob-jecti (in frame i) and shape class ci: dci,I • Transition costs occur for each change of the shape class: wci-1, ci • Solve the minimization problem:

24. Experimental Results standing walking turn around sit down sitting

25. Conclusions • New algorithms to classify postures and gestures of a person in a video were de-veloped at the University of Mannheim. • A major deficiency of the curvature scale space approach is the fact that convex regions of a shape are not represented in the CSS diagram. • We propose mapped shapes, mirrored at a circle around the object, to overcome this problem.