Dynamic Scenes by Image Sequence Analysis

1. Dynamic Scenes by Image Sequence Analysis Jun Shen 2004

2. Presentation scheme • General presentation • Dynamic scene analysis (DSA): a general view • Motion detection by background subtraction & by orthogonal moments • 3D-model-based vehicle pose determination & tracking • Face tracking • Gait tracking (Model-based tracking) • Automatic gait recognition • Learning & recognition of activity patterns by fuzzy self-organizing Kohonen net • Demonstration of results

3. I. General Presentation

4. . . . Camera 1 Camera n PROCESSING Environment modeling PROCESSING Motion segmentation Object classification Tracking Behavior understanding and description Personal identification Fusion of Information from multiple cameras General framework of visual surveillance I. General presentation

5. II. Dynamic Scene Analysis (DSA):A general view

6. II. Dynamic Scene Analysis (DSA):A general view Low-level analysis • Motion detection • Pose determination • Hidden effect processing • Moving object classification • Tracking II. DSA: A general view

7. Motion detection methods • Background subtraction • Temporal difference between successive frames • Optic flow • Matching: correlation, etc • Frequency domain methods • Pose determination II. DSA: A general view

8. Moving object classification • Classification based on geometric and radiometric properties of object • Shape • Size • Height-width ratio • Color • Texture • Features • Classification based on motion II. DSA: A general view

9. Tracking • Tracking based on regions • Tracking based on moving contours • Tracking based on features • Tracking based on object models II. DSA: A general view

10. Behavior understanding and description • Finite state automate • Non-deterministic automate • Hidden Markov process model • Neural nets • Syntactic methods …. II. DSA: A general view

11. Person identification by gait analysis for video surveillance • Model-based methods • Statistical methods • Characteristic-parameter-based methods • Temporal-Spatial-motion-based methods • Combination of gait analysis with other biometrics methods II. DSA: A general view

12. Fusion of information from multiple cameras • Positioning of cameras • Calibration of cameras • Matching of objects from multi-camera • Switching of cameras • Fusion of information from multi-camera • Hidden effect processing using multiple cameras II. DSA: A general view

13. III. Motion detection

14. III. Motion detection • Background subtraction • Temporal Gaussian-Hermite moments • Moments • Orthogonal moments • Gaussian-Hermite moments • Motion detection by Gaussian-Hermite moments III. Motion detection: 1. Background subtraction

15. III.1. Motion detection in color (or gray value) image sequence by background subtraction

16. Input image sequence Moving pixel detection Background image creation Filtering Moving objects detected Illumination change elimination Shadow elimination Labeling System overview Based on background subtraction III. Motion detection: 1. Background subtraction

17. False moving object Filtering & background image (c'tnd) • A mobile object stops during a period > half the temporal W. size, • It would be considered as static object and backgr'd updating will take moving object color. • When it begins to move again, backgr'd image thus updated would disturb the detection of its motion (double moving objects detected). III. Motion detection: 1. Background subtraction

18. Filtering & background image (c'tnd) • Solution • Color of moving pixels not taken into account in backgr'd updating. • Distinguishing stopped “mobile” objects from real static objects. • Comparison of present & preceding positions tells in motion or a stopped mobile object. III. Motion detection: 1. Background subtraction

19. R,B,G Channels of Difference Image Current image Difference image Diff Background image Motion detection by background subtraction for color images • Difference between current frame & backgr'd III. Motion detection: 1. Background subtraction

20. Motion detection by b'gd subtraction (c'tnd) • Difference between current frame & backgr'd • Segmentation of the difference color image • Fuzzy segmentation of R,B,G channels separately • Automatic determination of threshold T, • Fuzzy set “mobile pixels” by non-sym. p m'ship function. • Fuzzy segmentation with 3 channels together III. Motion detection: 1. Background subtraction

21. hi Threshold by "Max. Distance" Immobile Fuzzy set of mobile i T Motion detection by b'gd subtraction (c'tnd) • Difference between current frame & backgr'd • Segmentation of the difference color image • Fuzzy segmentation of R,B,G channels separately • Automatic determination of threshold T III. Motion detection: 1. Background subtraction

22. p(x) l2= + x T c= dmax+ k/( dmax- dmin), (k>0) dmax and dmin, max. & min. intensities. Motion detection by b'gd subtraction (c'tnd) • Difference between current frame & backgr'd • Segmentation of the difference color image • Fuzzy segmentation of R,B,G channels separately • Automatic determination of threshold T, • Fuzzy set “mobile pixels” by non-sym. p m'ship function. • Fuzzy segmentation with 3 channels together III. Motion detection: 1. Background subtraction

23. Actual color frame Background image Difference Image R channel B channel G channel ATD(Automatic Threshold by max. distance) ATD ATD Fuzzy M’ship fn Fuzzy M’ship fn Fuzzy M’ship fn Fuzzy deduction Automatic threshold by moment conservation method Mobile pixel image Fuzzy Segmentation of Difference Image III. Motion detection: 1. Background subtraction

24. Elimination of false motion due to illumination change • Problem • Bg'd image update using preceding frames not fast adapted to illumination v. • Rapidness of bg'd adaptation depends on temporal window size & bg'd adaptation method. • Even auto-adaptation used, bg'd adapted to illumination change only after an accumulation of frames III. Motion detection: 1. Background subtraction

25. Mobile pixels detected by background subtraction for the current frame Mobile pixels detected by variation in successive frames Mobile pixels detected in preceding frame OR AND Validated mobile pixels Diagram of false motion elimination III. Motion detection: 1. Background subtraction

26. Center of gravity Shadow Elimination • Problem: Shadows of moving objects being of almost the same motion as moving objects • Importanceof shadow elimination Obtaining more precise description of moving objects III. Motion detection: 1. Background subtraction

27. III.2. Motion detection by orthogonal moments

28. III.2. Motion detection by orthogonal moments • Moments • Geometric, Legendre & Hermite moments • Behavior in space & frequency domains • Gaussian-Hermite (G-H) moments • Motion detection by G-H moments • Comparison with other methods • Concluding remarks III. Motion detection: 2. G-H moments

29. Geometric, Legendre & Hermite moments and their calculation • Geometric moments and their calculation • 1D geometric moments Mn(x) at point x: • Mn(x)= S(x+ t) tn dt n= 0, 1, 2, ... • 2D geometric moments of a 2D image I(x, y): • Mm, n(x, y)= I(x+ u, y+ v) um vn du dv • Fast algorithms, such as Pascal Triangle. • Explicit statistical signification. • Functional analysis viewpoint: Signal projected onto polynom. space, taking monomial functions as bases. III. Motion detection: 2. G-H moments

30. Orthogonal Legendre moments • Using orthogonal bases: • Calculation could be reduced, • Error easier to estimate when limited proj. used, • Reconstruction simpler. • Orthogonal Legendre polynomials: • (dn/ dxn) (x2- 1)n / (2n. n!) for xÎ [-1, 1], • Pn(x) = { • 0 otherwise. III. Motion detection: 2. G-H moments

31. Scaled Legendre polynomials: [(dn/ dxn) (x2- w2)n ]/ [(2 w)n. n!] for xÎ [-w, w] Ln(x) = { 0 otherwise. • n-th order orthogonal Legendre moment: Mn(x) = S(x+ t) Ln(t) dt = <L0(t), S(x+ t)> (inner product). III. Motion detection: 2. G-H moments

32. Recursive calculation of Legendre moments • The nth order orthogonal L. moments, calculated from window [x- w, x+ w], can be computed from (n- 1)th & (n- 2)th order L.M. : M0(x) = <L0(t), S(x+ t)> = S1(x+ w) - S1(x- w) M1(x) = <L1(t), S(x+ t)> = [S1(x+ w) + S1(x- w)] -<L0(t), S1(x+ t)>/ w Mn(x) = <Ln(t), S(x+ t)> =<Ln-2(t), S(x+ t)>- [(2n- 1)/ w]<Ln-1(t), S1(x+ t)>, for n> 1 with S0(t)= S(t) and Si(t)= Si-1(y) dy for i= 1, 2, 3, … • Si(t) easily calculated from Si-1(t) by recursive sum-box tech. III. Motion detection: 2. G-H moments

33. 2D Legendre moments In 2D cases: kx ky Mp, q(x, y)= òò I(x+ t, y+ v) Lp(t) Lq(v) dt dv -kx -ky • Separable, decomposed into cascade of 1D calculation, by recursive algo. III. Motion detection: 2. G-H moments

34. Hermite moments • Scaled Hermite polynomial Pn(t)= Hn(t/ s) with Hn(t)= (-1)n exp (t2) (dn/ dtn) exp (-t2). • 1D n-th order Hermite moment: Mn(x, S(x))= Pn(t) S(x+ t) dt n= 0, 1, ... • 2D Hermite moments of an image I(x, y): Mp,q(x,y,I(x,y))= Hp,q(t/s, v/s) I(x+t, y+v) dt dv with Hp,q (t/ s, v/ s)= Hp(t/ s) Hq(v/ s). • Separable, calculated by cascade of 1D. III. Motion detection: 2. G-H moments

35. Behavior of geometric, Legendre & Hermite Moments in space & frequency domains • Importance of behavior analysis • Behavior in space domain • Behavior in frequency domain III. Motion detection: 2. G-H moments

36. Geometric moment base functions • Graphs of similar shapes, • Moments considered as projections onto base function space, not efficient for diff.spatial modes. • Hermite & Legendre mnt. base functions • Many oscillations, depending on the order, • Extract efficiently characteristics of diff. spatial modes (orthogonal polynomial of order n has n diff. zero-crossings). • Oscillations in Hermite bases much less important than Legendre ones (because the Hermite bases are not really orthogonal). • Same conclusion holds in 2D cases. III. Motion detection: 2. G-H moments

37. Geometric moment base functions: • low-pass kernel, FT monotonically decreased. • Hermite moment base functions: • as order increased, max. FT position moves to right, and more and more similar to a band-pass kernel. • Legendre moment base functions: • best band-pass characteristics except for very low orders. The higher the order is, the more to the right the pass-band moves. • L. moments separate characteristics in different frequency bands better than H. moments, which are in turn better than geometric ones. III. Motion detection: 2. G-H moments

38. Gaussian-Hermite Moments III. Motion detection: 2. G-H moments

39. Property of G-H moments III. Motion detection: 2. G-H moments

40. Comparison • G-H moments better separate diff. bands. • Larger quality factor Q= (Center freqency)/ (Effective bandwidth). • G-H moments & G.-filtered deriv.: • G-H moments: linear combinations of Gauss-filtered derivatives of signal. • Construct orthogonal features from Gaussian-filtered derivatives. III. Motion detection: 2. G-H moments

41. G.-H. moments & wavelet analysis • Derivatives of Gaussians widely used as mother wavelets, • Different order derivatives of Gaussian filters define different wavelets, • Derivatives filtered by Gaussian filters of different s represent the decomposition of signal into wavelets. • Smoothed orthogonal Gaussian-Hermite moments offer a solution to construct orthogonal features from the wavelet analysis results. III. Motion detection: 2. G-H moments

42. 2D orthogonal G-H moments III. Motion detection: 2. G-H moments

43. Performance comparison: Sensibility to noise • Noise-free images and noisy ones with additional random noise, • Moment vectors (m0,0, m0,1, …, m0,5, m1,0, m1,1, …, m1,5), • Normalized distances between noise-free images and noisy ones. III. Motion detection: 2. G-H moments

44. Orthogonality equivalence To better understand the good performance of orthogonal moments in both spatial and frequency domains, we have • Orthogonality equivalence theorem - Orthogonal moment base functions are not only orthogonal in spatial domain but also in frequency domain. III. Motion detection: 2. G-H moments

45. Experimental verification • Three different reference shape images: quadrilateral, hexagon and octagon. • Noisy images: adding random noises of diff. standard deviations. • Each shape characterized by 12 moments of orders (0,0), ..., (0,5), (1,0), ..., (1,5). Geometric, H. and L. moments are tested. • Classification by comparing moment vector of noisy shape with the 3 ref. shapes. III. Motion detection: 2. G-H moments

46. Motion detection by Gaussian-Hermite moments • Why using G-H moments • Motion detectionusing G-H moments • Resultsand comparison • Comparison with differential methods • Comparison with background subtraction • Comparison with adaptive background subtraction III. Motion detection: 2. G-H moments

47. Why using G-H moments? • Methods of motion detection in image sequence • Background-subtraction-based, including stochastic estimation of activation Difficulty • Frame-to-frame illumination changes, • Slowly moving and/or uninterested moving objects • Calculation of adaptive background images demanding accumulation of a large number of images. • Based on temporalvariation in successive images Difficulty • Sensibility to noise III. Motion detection: 2. G-H moments

48. Advantages ofusing orthogonal G-H moments for motion detection • G-H moments: linear combinations of image derivatives, permitting to detect image changes • Much smoother than other moments, therefore much less sensitive to noises, facilitate moving object detection in noisy image sequences. • Odd-order G-H moment base functions: linear combinations of odd order derivatives of Gaussian functions. • Temporal G-H moments: composed of temporal image derivatives to detect moving objects in image sequences. III. Motion detection: 2. G-H moments

49. Detecting moving targets using G-H moments of different orders Given an imagesequence • Calculation of temporal G-H moments M1, M3 andM5 • Fuzzy motion detection by moment image segmentation, using threshold by improved invariable-moment-method, using non-sym. p Mship functionforeach point in moment images. • Membership function update by fuzzy relaxation: spatial relation between pixels in single and successive frames • Moving pixel decision III. Motion detection: 2. G-H moments

50. Comparison with differential methods Comparison with other methods III. Motion detection: 2. G-H moments