Fahad Zafar , Dr. Yaacov Yesha , Dr. Aldo Badano IAB Meeting Research Report June 2013

1. Ideal Observer Approach for Assessment of Image Quality on Stereo Displays Devices for Medical Imaging. FahadZafar, Dr. YaacovYesha, Dr. Aldo Badano IAB Meeting Research Report June 2013 CHMPR IAB 2013

2. Introduction • Evaluation of Stereo Displays for medical imaging diagnosis. • Image quality assessment of stereo display devices for medical imaging applications. • Investigate • Fundamental limitations and benefits. • Parameters crosstalk, display noise, luminance, stereo technology. CHMPR IAB 2013

3. Introduction • The goal of this research is to assess image quality on stereo display devices for medical imaging applications which have great potential since they provide depth perception to the observer when looking at medical datasets. CHMPR IAB 2012

4. Objectives • Mathematical Model for Image Quality Assessment on Stereo Displays • Propose a Stereo Model Observer. • Computationally investigate parameters • Crosstalk, stereo angle, display noise. • Effect on performance related to • 3D display device. • 3D display technology. CHMPR IAB 2013

5. Objectives • Human observer studies are slow and costly, we propose to use a computational model that can conduct signal detection tasks. Using this model we can explore multiple parameters for the display with many different settings which is very hard to do with human observers. • Crosstalk: Luminance intended for one eye, leaking into the other. • Stereo angle: The angle between the two stereo cameras and the center of the focus point. • Display noise: Unintentional output luminance disturbance present in every display. CHMPR IAB 2012

6. Success Criteria • Present an Ideal Linear Stereo Observer model that can be used to assess image quality for 3D displays using computational approach. • Show applications of the stereo model to compare multiple 3D display devices with different technologies (active vs. passive) • Present another use case for the stereo observer that does not directly relate to medical imaging. CHMPR IAB 2013

7. Success Criteria • Show that the stereo observer we have formulated clearly assess image quality for different 3D display devices using a computational approach. • Compare different types of 3D display technologies using our model. We would also like to use the stereo observer for other assessment tasks not related to medical image. • One example is compression of a stereo stream and understanding the tradeoffs between quality and compression ratio. CHMPR IAB 2012

8. Existing Solutions • Human Observer Studies • Content • Use of entertainment 3D content and no connection to medical imaging tasks. • Scope • Different scope (target the observer, not displays) • Limited in scope. • Results • Lengthy and expensive. • Results are often subjective and inconsistent. • Too many variations in 3D display technologies and multiple parameters to explore (MIP vs. Absorption model). CHMPR IAB 2013

9. Existing Solutions • Current work does not associate directly with medical imaging applications. Human observer studies conducted mostly user visual datasets from video games and the movie industry. • The task performed by human observers in existing published research is focused on the user not the 3D display. We believe there are just too many parameters to explore and thus conducting a human observer study is infeasible. CHMPR IAB 2012

10. Development Approach: Ideal Linear Observer CHMPR IAB 2013

11. Development Approach: Ideal Linear Stereo Observer • gl, gr: pixel data in the left and right image respectively, after visualization. • Sv: Physical Display emulation operator. • Ll, Lr: Luminance reaching the left and the right eye respectively. • g: discrete dataset. (eg. Simulated White Noise generated through a Random Number Generator.) CHMPR IAB 2012

12. Development Approach • Mathematical formulation of the observer. We use Signal to Noise ratio to assess performance of the observer. Essentially, the data, which is a simulated medical imaging dataset is visualized using a rendering algorithm (Sg) and then post processing is applied to emulate the display (Sv). • SNR: Signal to noise ratio. • KLg-1: Inverse covariance matrix of the background. (background could be White noise or Lumpy (which is a bunch of Gaussians super imposed on each other) ) • S: signal image (3D Gaussian blob) CHMPR IAB 2012

13. Development Approach • Sg: The visualization operator. Could be Maximum Intensity Projection or Absorption Model. • Alpha: Transparency. CHMPR IAB 2012

14. Development Approach Previous Results No visualizati-on 2D/3D Observer Sg Our Results ? Sg Orthographic, Voxel/Pixel = 1, Opacity = 1, Vision = Absorption or MIP 2D Observer Sg Orthographic, Voxel/Pixel = 1, Opacity = 1, Vision = MIP 3D Observer Sg Orthographic, Voxel/Pixel = 1, Opacity = X, Vision = Absorption Stereo Observer CHMPR IAB 2013

15. Development Approach • With our approach you can emulate all sorts of displays including 2D, stereoscopic 3D, multi-view stereoscopic 3D etc. Our computational model is a generalized model that can be tweaked to any form of observer by simply changing a few parameters. • You can use MIP or absorption for a single trial. CHMPR IAB 2012

16. Development Approach • Vision = the vision model used. It could be either Absorption or MIP. We have only used Absorption so far in our trials. • Voxel/Pixel ratio: is set to one, so one voxel in the 3D dataset covers 1 exact pixel on the screen when viewing it head on. CHMPR IAB 2012

17. Development Approach • 2D ideal linear is from previous work, but we have generalized it using our approach and conducted experiments using it to compare with previous work. We compare and confirm that they are both equal. But, with our model we can swap parameters to do more observers like Ideal Linear 3D and stereo. So our approach is more robust • Opacity = X means opacity is varied while everything else is kept constant. Opacity is the opposite of transparency. CHMPR IAB 2012

18. Development Approach • We have results for 2D, 3D and stereo. 2D and 3D are computed only to compare and establish our approach as equivalent to previous work but no one has done stereo. Our model can do all these three and more (such as multi-view stereo observer represented by the ? sign) CHMPR IAB 2012

19. Development Approach • -- MIP - both the meaning of the acronym and the definition • Maximum Intensity Projection is a rendering model where we shoot rays from each pixel into the voxelized dataset. The ray traverses through all the voxels with in line of sight and the voxel with the highest intensity (or voxel value) with in that line of sight is selected as the final color of that pixel. CHMPR IAB 2012

20. Development Approach • Absorption model is an alternative rendering model, we shoot rays from the pixels and interpolate between the voxels as the line of sight intersects with them. While interpolating between voxel values, there is a weighting parameter applied to each voxel. That parameter is called the "alpha" or transparency.  So the equation looks something like this .. • color-voxel-1 * alpha1 + (color-voxel-2 * (1- alpha1)) CHMPR IAB 2012

21. Development Approach • And you do this for the first 2 voxels that the ray hits, then you traverse the ray and if it hits the third voxel, you use the previously generated color form voxel-1 and voxel-2 and interpolate with voxel-3 and so on. Sometimes you can fix the alpha for every slice of the volume dataset and every voxel has a fixed alpha. other times there are alpha maps that are generated with the dataset to accentuate some feature.. for instance one alpha map of a cube would highlight the lungs while another would highlight the heart etc.. but we dont do this. we use a constant for all voxel. CHMPR IAB 2012

22. Development Approach • Transparency is alpha • Opacity is the opposite of alpha.. .opacity = 1- transparency  CHMPR IAB 2012

23. Comparing 2D Ideal methodologies: Our method vs. Theoretical

24. Comparing 2D Ideal methodologies: Our method vs. Theoretical • So in this result, we perform the Ideal Linear 2D observer on the White Noise data. Our method outputs the exact same results as the theory suggests. • For white noise the outputs can be theoretically calculated so that how we know our SNR outputs are correct. CHMPR IAB 2012

25. Comparing 2D Ideal methodologies: Our method vs. Theoretical • The x-axis is the signal amplitude, meaning the amplitude of the Gaussian blob which the observer was viewing with in the image. • The y-axis is the final SNR. • These results do not emulate any display device and no crosstalk was simulated for these results. Adding crosstalk is our next step CHMPR IAB 2012

26. Our method quantifies stereo perception at different stereo angles

27. Our method quantifies stereo perception at different stereo angles • Now we use our model to calculate SNR at different stereo angles. No other method can potentially do that since they don’t visualize their dataset, hence cannot generate projections at different angles. • We quantify the different gain in observer performance when the stereo angle is changes. Beta = stereo angle in this case. CHMPR IAB 2012

28. Our method quantifies stereo perception at different stereo angles • The x-axis is the signal amplitude, meaning the amplitude of the Gaussian blob which the observer was viewing with in the image. • The y-axis is the final SNR. • These results do not emulate any display device and no crosstalk was simulated for these results CHMPR IAB 2012

29. Evaluation Resolution: 32x32 Number of Images: 240,000 Dataset size: 1 GB Covariance size: 2 MB 200 sec 900 sec 10 sec Total Simulation time = ~19 minutes • Successfully verified our model • Successfully implemented the simulation pipeline CHMPR IAB 2013

30. Evaluation • Currently, we have completed the implementation of the pipeline and it takes about 19 minutes for a complete trial which includes dataset generation, visualization (creating stereo pairs), calculating data statistics and then calculating the SNR • We compared 2D observer SNR for White noise with 2D observer SNR from previous works SNR and they were equal, within a range of <1% error. CHMPR IAB 2012

31. Evaluation • The equations applied are in the previous silde. 1. generate a dataset for some background (White noise or lumpy) 2. Then visualize using Absorption model volume rendering 3. Then create stereo pairs (about 240,000 image pairs). 4. Then calculate covariance for the images. CHMPR IAB 2012

32. Evaluation • 5. Then calculate inverse of the covariance matrix and multiply 3 things to get snr (signal image transpose * inverse covariance of background * signal image) • .6 proof of this methodology is present in the book “Foundations of Image Science” by: Harrison Barrett and Kyle Myers. CHMPR IAB 2012

33. Evaluation • Extend the model • Measure physical display crosstalk • Add additional stage to the pipeline and emulate the display • Conduct comparative analysis. Experimental setup Workflow CHMPR IAB 2013

34. Evaluation • Next, we are working on exploring the crosstalk parameter to see how it affects observer performance. We measure the crosstalk from actual 3D display device, experimental setup is presented in the slide. Then we use that data as a post processing function with in the model to emulate the display. • Crosstalk: Luminance intended for one eye leaking into the other. CHMPR IAB 2012

35. Evaluation • We measure the luminance response for both the left and the right displays independently through the 3D glasses. Then we measure the crosstalk through the glasses independently and combine all this data to create a luminance response for the display. This luminance response is used as a 2D lookup table with in the code and considered to be its emulation. • SNR calculated for the stereo observer for this 3D display device vs. SNR calculated without any display (which is the ideal since there is no display emulation). This comparision will show the quantitative decrease in perception because the device contains crosstalk and crosstalk negatively affects signal perception. CHMPR IAB 2012

36. Status • Have a working pipeline that simulates display devices. • Have collected luminance data from one passive 3D display device. • Next step • Emulate the device and analyze performance. • Get luminance measurements of active 3D displays and compare. CHMPR IAB 2013

37. Status • Have a working pipeline. We are currently trying to emulate an active and passive stereo device and compare their performance. We would like to measure luminance from many displays and then add do comparative analysis using our model. • We measure the luminance response for both the left and the right displays independently through the 3D glasses. Then we measure the crosstalk through the glasses independently and combine all this data to create a luminance response for the display. This luminance response is used as a 2D lookup table with in the code and considered to be its emulation. CHMPR IAB 2012

38. Thank You! CHMPR IAB 2012