Video Paintbox: A Novel Framework for the Artistic Rendering of Video

1. John P. Collomosse and Peter M. Hall MTRC, Department of Computer Science, University of Bath, Claverton Down, Bath (BA2 7AY) Introduction We present the Video Paintbox; a novel suite of software algorithms capable of transforming video clips into artistically stylised animations. Our automated system is able to render video into a wide gamut of visual styles, encompassing cartoon-shading, oil painting, sketching, even Cubism, and is also able to emphasise motion using traditional animation cues such as streak-lines, anticipation and deformation. As with drawings and paintings, animations are complex visual abstractions depicting significant (salient) details within a scene. We argue that comprehensive video analysis therefore forms a necessary first step in the artistic rendering (AR) process; salient information (such as object boundaries or trajectories) must be extracted prior to re-presentation in an artistic style. Prior to our work, AR algorithms performed a significantly lower level of analysis – for example, simply distorting small image regions to form brush strokes, and rendering frames independently. This resulted in animations of poor aesthetic quality and restricted style, often exhibiting a distracting flicker that required many hours of manual correction. By developing novel Computer Vision techniques for AR, we have mitigated many of these long-standing problems; for example reducing flicker in animations by an order of magnitude. Our initial results have been presented in leading international forums, and enjoyed recent media exposure both on broadcast television (CNN, DW-TV) and print (Times), in the latter case featuring a Cubist portrait of Charles Clarke MP. The next phase of work will investigate real-time video processing; in particular, the compact nature of animations shows potential in low bandwidth video-conferencing. We are also working towards immersive augmented reality environments and sketch querying for video databases, so continuing to contribute to the significant and developing convergence area between Computer Graphics and Vision. Cubism and oil painterly styles University of Bath Department of Computer Science Video Paintbox: A Novel Framework for the Artistic Rendering of Video Results High quality, flicker free rendering Cartoon motion emphasis Wide variety of artistic styles Hand-held DV camera System Architecture Motion emphasis requires accurate measurement of object motion parameters, and is complicated by the fact that camera motion must be compensated for. Camera extrinsics are estimated via a RANSAC based correlation technique (below left). Some motion cues e.g. squash and stretch, require detection of events in the video such as collisions. These are detected by learning the motion parameters of objects over time, and watching for “unexplained” changes in those parameters (below right). We have designed the Video Paintbox to give animators a high degree of expression over the character of their output by providing control over visual elements; but importantly also by providing a degree of automation that frees them from laborious work. For example, an animator might select a bouncing ball and specify “watercolour” and “squash-and-stretch”. The system would then take care of the details; the individual brush strokes, or the mechanics of deformation as the ball impacts upon the floor. The enabling factor for such automation is the tight coupling of Computer Vision and Graphics techniques within our system. Source Image Sequence Motion compn. and tracking Feature depth recovery Time and pose Motion emph. Visual motion emphasis Output Animation tracked features video (for occlusion det.) video AR shaded animation Stroke Surface framework Video Paintbox (Motion and Shading Subsystems) The Video Paintbox rendering pipeline comprises several sequential processes (above), centred around two subsystems; one for emphasising motion, and the other for visually stylising (shading) the video content. Coherent Shading The uniquely high level of temporal coherence afforded by the Video Paintbox is due to a novel spatiotemporal approach to processing video, dubbed “Stroke Surfaces”. In a spatiotemporal view of video, one considers the video as a 3D cuboid of pixels, X, Y, and time as the third dimension (see below). The Stroke Surface framework was first published in 2003, and was the first automatic AR video technique to process video in this way (after earlier interactive work by Klein et al.). Existing automatic techniques of the day (due to Litwinowicz, Hertzmann, and Kovacs et al.) processed video in a per frame sequential manner, painting the first video frame and moving strokes from frame to frame according to an inter-frame motion estimate. Such estimates can never be perfect in the general case, and so errors in local motion estimates accumulate and propagate throughout the video sequence. These errors are manifested as a distracting and unsightly flicker in the animation. By contrast the higher level spatiotemporal analysis performed by the Stroke Surface framework performs a global relaxation over the spatiotemporal video volume to smooth errors in the tracked trajectories of objects. In many cases this results in virtually flicker free video, in fact typical results exhibit around an order of magnitude less flicker than those produced by the prior state of the art. Motion emphasis • The motion emphasis subsystem enables the insertion of traditional cartoon motion cues into video. For our purposes we have classified these cues into three groups: • Augmentation Cues: Marks made to depict motion, e.g. streak-lines or ghosting. • Deformation Cues: Deformations performed on objects, e.g. squash and stretch. • Dynamic Cues: Effects which alter the timing of an animation or the pose of objects, e.g. anticipation or ``motion cartooning'' (motion exaggeration). • Processing proceeds as follows: • Tracker is boot-strapped via user initialisation of contours, which are tracked using a Kalman filter under affine motion constraints. • Occlusions are detected and a partial depth ordering is obtained via analysis of object occlusion graph. • Entire video is analysed prior to rendering, to determine salient movements within clip. • Video background and features are extracted from the video to create ‘cels’ in a virtual animators rostrum. • Feature cels are repositioned, warped and additional cels inserted, to create dynamic, deformation and augmentation cues respectively. • Finally, cels are re-composited (right) • Processing proceeds as follows: • Video frames independently segmented into regions • Regions in adjacent frames associated via heuristics • Undesirable features in association graph are culled e.g. short cycles or branches. • Spatiotemporal volumes formed by the associated regions are smoothed. • When intersected with a plane of constant time, regions are produced with smoothly varying boundaries