G . Aielli 1 st EDUSAFE technical Meeting - Tuscania 5/6/2013

1. WP2: AR new technologiesstatus and plansG. Aielli for the EdusafecollaborationCERN Kick off meeting 29/10/2013 G. Aielli 1st EDUSAFE technical Meeting - Tuscania 5/6/2013

2. WP2 Implementation summary Roma2 is responsible for WRM chip development (WRM chip and algorithm design, fast prototype development). EPFL will develop the 3D tacking system and adapt it to the WRM technology. NOCA will research the video and sensing data treatment and adaptation to computer vision requirements as well as wireless transmission. CERN is responsible for providing the environmental and usability requirements and developing the authoring and visualizer software prototype optimized for AR technologies and methods. Milestones:

3. The PTU implementation AT TUSCANIA TECH. MEETING Basic scheme of the system now we are much beyond ! server PTU user interface and visualization software Radio protocol slow monitoring HMD driver Tracking calculation DAQ WRM Standard Wi-fi Animation server Supervision system A/V Camera+ edge detection Guaranteed B.W. Wi-fi dosimeter Sensor box Should be mechanically locked HMD+audio Cable / wifi driven Based on the camera 3D pose calculation scenario (no object detection). A limit pointed out: the RT available edge detected image is too rough for implementing highly sophisticated algorithms  gray shades image is needed which is difficult to be achieved through WiFi in RT

4. constraints and Expansion of the basicscheme • Integration of WRM in the mobile system is out of EDUSAFE scope, a stage of electronic image processing can be integrated at camera board level • We assume we can not have full image Wi-Fi transmission in RT • Tracking and Object identification on the Mobile System : • ADVANTAGE wired transmission between camera and PTU fast • DISADVANTAGE  Computing power limitation  algorithm must be not resources hungry • CONSEQUENCE  Mobile visual tracking needs support from other systems in parallel (e.g. IMU sensors; parameter stream from WRM – see later) • Tracking and Object identification on the Server: • ADVANTAGE: no special computing power limitation; easy to integrate the WRM board on XPCI expansion slot (for example) • DISADVANTAGE: limited Wi-Fi transmission speed in RT (by now…) – only 1-2 bit-planes • CONSEQUENCE: WRM can extract rapidly features from the reduced frames and send the results to the algorithms running on the mobile system

5. Video / audio (Noca) 1-1 EDGE detection (Roma) 1-2 1-3 RT WiFiTx (Noca) The AR pathway – a cooperative scheme 2.1 WRM (Roma) Head mounted Data acquisition SERVER EDGED image ~ 1-2 Mbit/frame 2 1 Inertial sensors (Noca) 1-4 2 RT WiFi Rx 50-80 Mbit/s WRM feedback WRM Tracking software. 2.2 Parameter space Interface protocol (Noca) FullImage 24 Mbit/frame IMU data ~ kbit/s 3.1 PTU 3 RT WiFi Rx RT WiFiTx Parameter stream ~kbit/s 2D/3D tracking Visualization SW (CERN) • A cooperative/complementary approach (thanks to Marzio for the hint ) • WRM quickly calculate as much as possible pose and object parameters on limited data set • PTU tracking receive the WRM input and complete the job on the full data sample • IMU sensors backup the black-out frames and compensate for the visual tracking delay

6. Brief description of the trackingalgorithm EARLY STAGE DEVELOPMENT • The original tracking strategy was based on fast key-frame identification (through WRM) + calibration based on dense approach . • not optimal for the “maintenance scenario”  it requires a fixed background THE NEW STRATEGY • It is conceived for an highly changing environment ant it focuses on object detection. Here we implemented the cooperative approach described before • ON THE PTU • Probabilistic boosting trees = learning method for fitting regression models. We want to use PBTs for inferring the pose as a function of the appearance of the object. • The problem can be reduced by outsourcing to WRM some of the computing job. This is provided by the parameter stream • Also IMU sensors cooperate backing up the camera tracking (sensor fusion). For object detection they don’t help • ON THE WRM • R,G,B edge detected image is searched for patterns in the parametric space given by specific objects. A partial relative pose parameter extraction is attempted • At the same time the differential frame analysis can performed on the whole frame (differential visual tracking)  fast estimation of the camera movement

7. Video / audio (Noca) 1-1 EDGE detection (Roma) 1-2 1-3 RT WiFiTx (Noca) Building block scheme of the tracking functions 2.1 WRM (Roma) 2.7 SW FeaturesDatabase (EPFL) 2.8 WRM FeatureDatabase (Roma) Head mounted Data acquisition 1 Inertial sensors (Noca) 1-4 SERVER 2 RT WiFi Rx Visual Tracking software. Object and pose determination (Roma / EPFL) 2.2 Parameter space 2D objectdetection (EPFL) 3.3 3D pose determination (EPFL) Interface protocol (Noca) FullImage 24 Mbit/frame IMU data ~ kbit/s 3.2 3.1 RT WiFiTx Sensor Fusion (CERN) PTU 3 Supervision / Control and DAQ system (IASA) Grey Image 2.3 RT WiFi Rx 3.4 Visualization SW (CERN) 4 OPERATOR

8. THE WRM PRESENT Implementation • This is the first design proposed for the EDUSAFE front-end. • The image is taken directly from the CCD, transformed to edge detected image using a board implementing the zero crossing edge detector, which will be developed for the project. Two WRM chips and a local data rotation are needed to cover all possible angle coefficients. Three will be optimal (see ALI slides) • The vertex detector has the role of detecting edge crossing directly from the data parametric spaces produced by the WRM. • Specific edge crossing patterns can be used for object detection and evaluation of the camera pose

9. Interaction of the tracking and the visualization system • The WRM + the tracking codes running in the server and on the PTU will produce a stream of pose or object estimate • First of all to be meaningful both visualization and tracking must be synchronized to the user interface by the user commands: the user defines the working phase (scene) and triggers: • Appropriate upload of animation sets on the PTU • Appropriate pattern and methods to the tracking The AR is organized in scenes and the action is user command driven User interface Logic blocks: The real process can be executed part on the server part on the PTU Preload pattern Preload and cache animation HMD+audio User commands server Visualization system (on PTU) Tracking stream Rendered animation Tracking system

10. Primary TEST and DEV strategy WRM board WRM board server camera camera CPU (tracking process) CCD EDGE I/O driver NET (WRM) • What to do in wait of the RT WiFi and a running fast PTU? • Components of the system • Chip NET existing • Chip EDGE, I/O driver to be done, WRM board to be done • TEST of this architecture • MATLAB/C++ simulation infrastructure done, intensive test campaign ongoing • Low frequency HW test of NET done • High frequency HW test of NET end of the year • Design and simulation of EDGE an • d I/O driver 2014 • Final prototype test 2015 CPU (tracking process) EDGE I/O driver NET (WRM) CCD

11. Video / audio (Noca) 1-1 SERVER EDGE detection (Roma) 1-2 1-3 2 Head mounted Data acquisition RT WiFiTx (Noca) 1 The full picture 2.1 WRM (Roma) 2.7 SW FeaturesDatabase (EPFL) 2.8 WRM FeatureDatabase (Roma) Binary image RT WiFi Rx Inertial sensors (Noca) 1-4 Parameter space Full Color Image IMU data Standard WiFi and Router Visual Tracking software. Object and pose determination (Roma / EPFL) 2.2 2.6 RT WiFiTx 2D objectdetection (EPFL) 3D pose determination (EPFL) Interface protocol (Noca) Grey Image RT WiFi Rx 3.3 3.2 3.1 3.5 Supervision Sensors (Prisma) Sensor Fusion (CERN) IMU data Compressed color image Supervision / Control and DAQ system (IASA) PTU 3 2.3 3.4 3.6 WIFI module Transmission protocols (Noca) Supervision exchange Fused pose data 2D /3D objects proceduresDatabase (CERN) AR Content Visualization SW (CERN) 4 Gamma Imaging camera and algorithm (Canberra) 5 2.5 HMD 4 Display Hardware (TUM) 4.1 User interface (TUM+CERN) 4.2 Eye Trakcking (TUM) 4.3 2.4 Authoring tool (CERN)