Active Visual Observer Integration of Visual Processes for Control of Fixation

1. Active Visual ObserverIntegration of Visual Processes for Control of Fixation KTH (Royal Institute of Technology, Stockholm)and Aalborg University C.S. Andersen and H.I.Christensen

2. Architecture for controlling an agile camera • Basic system facilitates three low level processes: • Fixation • Tracking • Attention selection and shifting

3. The basic idea • Is that a tight coupling between the lowest visual processes, referred to as the basic system, and the sensing apparatus, with known latencies, is imperative for successful operation in dynamic environments. Following the biological inspiration, the basic functionality of a camera head are: fixation, gaze shift and smooth pursuit. • A system capable of addressing these aspects of active vision will be capable of fixating on an object, and maintaning fixation while it is moving, or during ego motion of the head.

4. The basic system layout

5. The attention mechanisms • The attention mechanism will allow for selection of interesting (salient?) points from the input data. The system can perform selection of fixation points, fixation and tracking. • Below is a standard control system for a DC motor with tachometer feedback , with “normal” appearance at the top and the control schematic at the bottom.

6. Standard control system

7. Designing the Architecture • Biologists have argued convincingly that eye movements typically are performed in two separate stages, Version and vergence, with both eyes participating in both motion patterns, while fixating at some point in the space.The version angle is the direction of gaze for an imaginary eye positioned between the two rotation centers in next figure.

8. Layout of Aalborg head

9. Cyclopean representation The version angle relies on the two vergence motor settings.The pan motor contributes however, along with the vergence motor to the direction of gaze.

10. Control • We may use one camera as leading and the other following. • The visual process of tracking in the leading eye approach is roughly equivalent to performing control of version and tilt in the cyclopean representation, while fixation corresponds to the process of vergence control. Hence renaming the modules and utilizing a different representation the basic control architecture may facilitate equal eye control as shown in figure below

11. Architecture for an equal eye dominance control scheme.

12. Notes to the figure • It should be noted that the figure only displays the “forward control lines” . Actually there are feedback signals form the hardware to the visual processes, as well as communication between the individual processing modules. The signals in the system is as described earlier the actions issued by the processing modules, which in this case is vergence,version and tilt angle adjustments. Thus the close connection with the actual control of hardware is still maintained

13. Completing the Architecture • So far we presented only the mechanical control associated with the eye movements. The system has addiotnal rotational degree of freedom, the pan. There is alos motorized lenses with 3 degrees of freedom: focal length (zoom), focus (accommodation) distance and aperture.

14. The modified cyclopean control architecture

15. An Experimental System • Final system relies on correlation based stabilization fro the left and right camera. The computed image slip from the two cameras is combined to form the error signal for control of the version and tilt angles. While a disparity estimate could be computed from the target location in the image pair, it has been chosen to perform an explicit disparity extraction by correlating the images. This provides redundant information but it also allows for a more robust control since a loss of disparity information does not necessarily mean that version and tilt control cannot be performed and vice versa.

16. Fixation distance for combined disparity and accomodation control

17. Attention selection • The figure below shows how the system selected areas of high contrast. Using the centroid of the receptive field as fixation point, the fixation has been shifted resulting in vergence-version-tilt angle changes as shown to the right of the figure below.

18. The receptive fields

19. Another Active Visual Observer • Binocular Active Vision System that can attend to and fixate a moving target, in particular is capable of figure-ground segmentation. This work focuses on occlusions of other both stationary and moving targets and integrate three cues to obtain an overall robust behavior, ego-motion, target motion and target disparity.

20. Major parts of the current system • Selecting a target • Control of the system for saccade and pursuit • Measuring the speed of the target for pursuit • Measuring and selecting a disparity for pursuit

21. System description • Fundamental skills are: fixation, target pursuit and target discrimination • The full system includes the integration of three cues for target selection and target discrimination. These are used by the moving observer to smoothly pursue moving or stationary targets binocularly while maintaining vergence. Mechanisms for discovering moving targets provide means of attention. There is another mechanism to find and select new locations to attend to.

22. The system implementation schema (the diamond indicates one frame delay in the feedback)

23. Motion detection schema

24. The Algorithm • Affine Background Motion Model is used for fit. Two steps involving feedback are included to account for object motion and large background motion. • The predicted and previous position and extent of the target is used to mask out parts of the image which likely belong to the object, so that they do not affect the calculation of the affine parameters for the background. • The accumulated parameters are used over time to cancel out the majority of the time difference, see feedback into WARP. • Background segmentation makes a use from the affine calculations

25. Target segmentation • The aim is to determine which parts of the scene are moving consistently with what is currently believed to be the target. • The calculations on the target are performed in analogy with what is done for the background motion ,i.e. affine model is used.

26. The target segmentation

27. Disparity segmenttaion

28. Disparity selection • The object of disparity selection is to select the disparities that belong to the target in the presence of disparities that arise from other locations in the scene. They are using the disparity histogram ,selecting the highest peak.

29. Integration • Target areas that do nto get support from either motion detection or taret segmentation are excluded from the target model. Also the disparity module detects areas in the scene that lie in front of the pursuing target, which are then excluded from the target model. • Areas that are detected as both moving independently as detected by motion detection and are moving consistent with the target image velocity model from the target segmentation are added to the target model.

30. The experimental platform

31. Real time pursuit • Centers a target visually while the target is moving across the room. In the second figure when the target in last row second frame moves behind the occluding object the pursuit does not follow the target,but stays on the occluding object

32. Real time pursuit

33. Real time pursuit

34. Figure ground segmentation extracts the target from previous sequence

35. Figure ground segmentation,cont.

36. Motion detection returns areas that possibly belong to moving target

37. Traget segmentation returns areas that are believed pursued target

38. Pursuit, can handle occlusion

39. Traget pixels extracted from the previous sequence

40. Target pixels extracted without disparity cue. Attention shifts to the second moving perosn

41. Pursuit during target expansion

42. Notes • On the top is shown the original sequence during pursuit. • The bottom row shows result of the segmentation every 3d frame

43. Pursuit of a rotating umbrella

44. Motion detection with real time motor control feedbacl

45. Tracking the white box