Presentation of Vista team (VIsion Spatio-Temporelle et Apprentissage)

1. Presentation of Vista team(VIsion Spatio-Temporelle et Apprentissage) Patrick Bouthemy UR Rennes, IRISA (http://www.irisa.fr/vista/)

2. VISTA-2005 research staff INRIA P. Bouthemy, F. Cao (2001), I. Laptev (2005), P. Pérez (2004), C. Kervrann(2003, on Inria secondment, Inra) CNRS J. P. Le Cadre University of Rennes 1 E. Mémin J. Yao (scientific collaborator, Maths dept., Irmar) Secretary:H. Béchu (Inria, with Temics team)

3. VISTA-2005research staff– continued • Ph-D students (11) G. Piriou, T. Bréhard, T. Veit (tbd, Dec. 2005) V. Auvray, A. Cuzol, J. Boulanger (3rd year) A.Bugeau, N. Papadakis (2nd year) T. Crivelli, C. Simonin, A. Hervieu (1st year) • Post-docs (2) I. Laptev (until August 2005), Venkatesh Babu (until March 2005) • Temporary technical staff (3) N. Gengembre (FT-RD contract), P. Heas (IST-FET Fluid), B. Fauvet (RIAM Feria project, until June 2005)

4. Research topics (1) • Overall objective: Analyzing dynamic scenes or physical phenomena from image sequences (video, IR, satellite images, video-microscopy) • Involved topics: • Restoring image sequences • Detecting, segmenting, tracking moving entities • Measuring and representing image motion • Modeling, learning and recognizing spatio-temporal contents • Problem formulation based on visual (2D) motion models and measurements

5. Research topics (2) • Applied maths foundations: mostly statistical approaches for image analysis • Probabilistic models (Markov models,probabilistic mixed-state models) • Bayesian inference, robust estimation, non-parametric estimation • A contrario decision methods • Particle filtering • Robust classification

6. Application domains • Content-aware video applications (multimedia and TV, sports videos, enriched video, video summarization, video indexing, video re-purposing) • main partners: INA, Thomson, FT-RD • projects: IST Lava (XRCE, Lear team), RNTL Domus Videum (Atlas,Metiss and Temics teams), RIAM Feria (Texmex team), IST-NoE Muscle (Ariana, Imedia and Texmex teams), FT-RD contract, ACI Behaviour (UTC, PSA) • Experimental fluid mechanics and Meteorological imagery • partners: Cemagref, LMD • projects: Eumetsat, IST-FET Fluid, ACI Assimage (Clime and Idopt teams) • Biological imagery (video-microscopy) • partners: INRA, Curie Institute, Univ. Rennes 1 (Dept of Biology) • projects: ACI-IMPBio MoDynCell5D, AC DRAB

7. Main evolutions 2001- 2005 • Important changes in the team composition • Creation of Texmex team (2002, P. Gros, 2 people from Vista, multimedia indexing, as announced in 2001) • Creation of Lagadic team (2004, F. Chaumette, E. Marchand, F. Spindler, 11 people all from Vista, as announced in 2001) • Creation of Visages team (2004, C. Barillot, P. Hellier, S. Prima, 6 people from Vista, medical imaging) • Arrival of 4 research scientists over 2001-2005 (3 Inria, 1 Inra) • Given the present Vista team composition and the corresponding re-focusing (in 2004) on dynamic scene analysis: • Achievement of (or at least investigation of) all the objectives listed in 2001 (apart from a minor point, NDC of structures under strain) • Addition of two new topics (a contrario motion analysis, video-microscopy) and reinforcement of a third one (video analysis and tracking).

8. Main contributions (1) • Adaptive non-parametric estimation for image sequence denoising • Efficient method based on locally adaptive kernel regression while preserving space-time discontinuities without motion estimation • Data-driven adaptive estimation window based on bias-variance trade-off criterion • Point-wise and patch-based versions • Experimental comparative evaluation on usual video image sequences: our method outperforms other recent methods(joint Kalman-Wiener, wavelet-based, and PDE-based methods) Left: original sequence; middle: noisy sequence; right: denoised sequence with our method

9. Main contributions (1bis) • Application to video-microscopy • Goal: detection and tracking of moving intra-cell protein vesicles to model and interprete their dynamics for biological studies • First issue addressed: 3D image sequence denoising • Positioning (national and international context):Pasteur Institute (J.-C. Olivo-Marin), ENS Paris, EPFL (M. Unser), Heidelberg Univ. One 3D image of the sequence: noisy (up) and denoised (bottom) Image sequence provided Denoised sequence by Curie Institute, with our method study of Rab6 protein

10. Main contributions (2) • A contrario methods for motion detection and matching • Decision is performed a contrario: large deviation (very small probability, NFA computation) to a randombackground model (independence assumption), no prior required on the searched entities, importance of the measurement choice. • General, simple and parameter-free (automatic threshold setting) method while supplying confidence values. • Pos.: ENS Cachan (J.-M. Morel,a contrario framework for static image analysis, collaboration on shape matching and recognition),Ceremade (F. Dibos), UPF Barcelona (V. Caselles)

11. Main contributions (2bis) • Addressed problems • A contrario instantaneous detection of moving regions • A contrario coherent motion detection (clustering in an appropriate space, initialization for tracking) • A contrario image comparison (global and local criteria) and retrieval in video streams

12. (false detection) Main contributions (3) • Mixed-state probabilistic models for motion modeling and recognition • A unique representation for information which can be both discrete (symbolic) and continuous. No supplementary (hidden) label field requiring an inference stage. • Definition of specific mixture distributions (Dirac + exponential family) to model (global or local) occurrence statistics of 2D motion measurements for motion classification, supervised motion recognition and event detection(in sports videos).Recognition of serve shots • Definition of mixed-state auto-models(extension of Besag’s framework to multi-parameter cases) and first application to the modeling of temporal (motion) textures. • Pos.: UCF (M. Shah), Weizmann Institute (M. Irani), Microsoft Research (H.-J. Zhang), Univ. Rochester (M. Tekalp), UCLA (S. Soatto, linear models for dynamic textures)

13. Main contributions (4) • Fluid motion analysis • Estimation of dense velocity fields for fluid flows • Specific data model (continuity equation) and second-order div-curl regularization • Parsimonious decomposition on adapted basis functions (vortex and source particles) • Estimation of the motion of multiple layers (exploiting a prior image segmentation) • Intensive evaluation on typical fluid flow experiments (in collaboration with Cemagref) • Pos.: Correlation-based methods in applied meteorology (wind field computation), PIV (Particle Image Velocimetry) techniques in experimental fluid mechanics (Image sequence provided by Onera)

14. Main contributions (4bis) • Fluid motion analysis (continued) • Structuration of the fluid flow field • Estimation of the potential functions associated with the irrotational and solenoidal components of the flow field (Fourier and variational approaches) • Tracking of high-dimensional flow structures (e.g., motion fields) • Stochastic filtering relying on the vorticity-velocity form of Navier-Stokes equation and then Itô diffusion process • Deterministic filtering exploiting a variational data assimilation technique Tracking of a cyclone by stochastic filtering 16 vortex particles 1000 filtering particles

15. Main contributions (5) • Non-linear tracking methods • Robust tracking in low-dimensional state-space and complex likelihoods • Multi-object tracking with particle filters (novel SMC techniques, data-target association issue) • New conditional formulation of classical filters with state equation estimated from the image data (point and planar structure trackers) • Tracking with auxiliary variables (handling of visibility, occlusion events) Left: our method (item 2), all the points are correctly tracked, Right: STK tracker (several points are lost) Tracking with auxiliary variables of visibility

16. Main contributions (5bis) • Non-linear tracking methods (continued) • High-dimensional state spaces and detailed dynamics • See previous slide on fluid motion analysis • Target tracking for partially observed non-linear systems • Initialization of particle filtering based on a hierarchical approach • Derivation of the PCRB closed-form (Posterior Cramer-Rao Bound) for performance analysis and sensor management • Design of decentralized particle filters • Pos.: Georgia Tech (F. Delleart), EPFL (P. Fua), ETHZ (L. van Gool), Brown University (M. Black), Siemens Princeton (D. Comaniciu), Microsoft Research (A. Blake), DSTO Adelaide (N. Gordon), Qinetiq UK (S. Maskell) Distributed target tracking with two observers(in blue) True trajectory in green, estimated trajectories obtained by each observer in red. Confidence areas for each observer in red are large. Distributed target tracking in black. Confidence area in black is very small.

17. Some highlights (1) • Academic dissemination and visibility • 16 Ph-D theses, 62 journal papers (2001-2005), 2 books, 2 best paper awards • Co-general chair of ICCV’07, organization of CBMI’03 (of Miccai’04, Visages team), PC co-chair of RFIA’06, Associate editors IEEE-IP, Area editor JAIF. • Involvements in industrial partnership, applications and transfer • Consortium (GIS Aeternum Multimedia) formed by Thomson, INA and Irisa (Metiss, Texmex and Vista teams) • Cross-modal video summarization tool validated on football and rugby TV programs (Thomson) • FT-RD contracton team sports video processing (Rugby world cup’2007) • Head of European FLUID project (LaVision PIV company); transfer of estimation tool of dense fluid flow fields to meteorological domain (Eumetsat) and experimental fluid mechanics one (Cemagref, Onera)

18. Some highlights (2) • International collaborations • Associate Inria team FIM (University of Buenos-Aires, on fluid motion analysis and on dynamic texture analysis) • Univ. of Mannheim (C. Schnörr), Univ.of Cambridge (J. Vermaak), Univ. of Las Palmas (L. Alvarez), UCSD (S. Belongie), European NoE Muscle • Software • Motion-2D (source code for estimating parametric motion models, publically available under QPL): about 550 downloads (explicitly identified) since August 2003(http://www.irisa.fr/vista/Motion2D/) • Tracking application (on-going development) • Platform • Multimedia platform (with Texmex and Metiss teams, collaboration with INA, starting 2005): capture of TV programs, content server, metadata server, multimedia processing, content-based multimedia indexing, experimental evaluation,…

19. Perspectives (1) • Video analysis and understanding • Analysis of motion textures with mixed-state models (segmentation, classification) • Modeling and recognition of dynamic video contents from bags of trajectories • Video alignment (dynamic multi-view constraints) for content recognition and retrieval • Combined motion and object categorization and recognition • Learning issues: semi-supervised learning with tracking for video contents; dynamic kernel-based methods • Specific focus on sports TV programs (cross-modal analysis, self-structuration, video summarization)

20. Perspectives (2) • Tracking problems • Visual tracking with no prior (on-line learning and updating of appearance model, cluttered background, combination of complementary representations, conditional Markov models) • Camera network (partially observed dynamic system, low-level measurements, polyedral separation methods, trajectory reconstruction, situation analysis for groups of moving entities) • High dimensionality in visual tracking problems (variational assimilation techniques and non-linear Bayesian filters) • Trajectory optimization for terrain-aided navigation (collaboration with Onera)

21. Perspectives (3) • Image sequence analysis for environmental, fluid mechanics and biological studies • Further incorporation of physical laws • 3D motion estimation from 3D volume data or from multi-view 2D data (in experimental fluid mechanics and meteorological applications) • New involvement in oceanography applications • Continuation of our work in video-microscopy on the detection and tracking of small moving intra-cell entities • Interpretation of analysed dynamic contents in relation with biological studies (partnership with biologists) • Envisaged proposition (in 2-3 years) of a new joint Inria-Inra team devoted to biological image sequence analysis and related applications