Driver’s View and Vehicle Surround Estimation using Omnidirectional Video Stream

1. Head Detection & Tracking Face Orientation Estimation Driver’s View Generation Viewing Direction Pan/tilt angles to camera Head Detection & Tracking Face Orientation Estimation Driver’s View Generation Omnicamera Sub-sample and Grayscale Head Candidate Extraction Update Kalman Filter for Head Tracking Edge Detection Equalization To Face Orientation Estimation Perspective Transformation on Driver’s Seat Current frame of the image, with estimated image motion in the area of interest. Points used for estimation of ego-motion. Gray: inliers, White: outliers, Black: unused. Predict Head Location in Next Frame Constrained Ellipse Detection (RHT) Face/Non-face Classification (DFFS) Omni-Video Stream Ellipse Search Window CAN Bus Calibration Flat-Plane Transform Motion Transform Parameters Ego-Motion Compensation H Delay Inverse Flat-Plane Tx. x- , P- x, P Motion Parameter Correction Head Tilting Compensation Spatial/Temporal Gradients gx, gy, gt Orientation Normalized frame difference in the area of interest. Output after post-processing and clustering. Head Detection & Tracking View-Based Face Orientation Likelihood Fns. Kalman Filter ML 360 degree Normalized Frame Difference 180 degree 0 degree Post Processing & Clustering Obstacle Positions Head Tilting Compensation State Sequence 1 2 N Head Detection & Tracking Projection into Feature Subspace Direction of driver Direction of car 0 degree of camera Bayesian Correction to Motion Parameters • Approximate motion parameters obtained from calibration, CAN bus. • Planar motion compensation equation: • Optical flow constraint satisfied under favorable conditions: • Image motion is expressed parametrically in terms of motion parameters for a number of image points as: • Correction performed by update similar to iterated extended Kalman filter: Gaussian Likelihood Functions 1 M Kohsia S. Huang, Mohan M. Trivedi, and Tarak Gandhi khuang@ucsd.edu, mtrivedi@ucsd.edu, tgandhi@ece.ucsd.edu Computer Vision & Robotics Research (CVRR) Laboratory University of California at San Diego La Jolla, CA 92093-0434 Driver’s View and Vehicle Surround Estimation using Omnidirectional Video Stream Results Abstract Our research is focused on the development of novel machine vision based telematic systems, which provide non-intrusive probing of the state of the driver and driving conditions. In this paper we present a system which allows simultaneous capture of the driver's head pose, driving view, and surroundings of the vehicle. The integrated machine vision system utilizes a video stream of full 360 degree panoramic field of view. The processing modules include perspective transformation, feature extraction, head detection, head pose estimation, driving view synthesis, and motion segmentation.  The paper presents a multi-state statistical decision models with Kalman filtering based tracking for head pose detection and face orientation estimation. The basic feasibility and robustness of the approach is demonstrated with a series of systematic experimental studies. Keywords: Driver head tracking, face orientation estimation, driver’s view generation, surround vehicle detection. Research Objective Accurate and real-time estimation of driver’s face orientation, driver’s view, as well as vehicle surround for a driver assistance system. Driver’s View Generation Driver Head Detection and Tracking Surround Vehicle Detection Surround Vehicle Detection Estimation of Head Pose and Face Orientation Scheme 1 Scheme 2 (Future) Average Performance Head Detection Computation of Driver’s Viewing Direction Face Orientation Summary • Simultaneous driver head detection, driver face pose estimation, driving view generation, and surround vehicle monitoring in real-time using a single omni-video stream. • Suitable for novel televiewing interfaces, driver assistance systems, and driver distraction studies.