Real-Time 3D Image Segmentation and Spatial Rendering

1. Real-Time 3D Image Segmentation and Spatial Rendering

2. Contents • Overview of Project • Challenges • Learning SpatialGL • Storing/Retrieving Image Data • Displaying Image • Image Orientation • Colour Map • Conclusion • Further Improvements

3. Where We Began Image Segmentation 3D Reconstruction

4. Where We Ended Up

5. Challenges

6. Challenges – Learning SpatialGL • Challenge: SpatialGL has poorly documented API • Solution: Trial and error • Challenge: SpatialGL requires knowledge of advanced OpenGL concepts • Solution: Created small OpenGL applications to understand OpenGL

7. Challenges – Storing/Retrieving Image Data • Challenge: C++ does not allow dynamic multi-dimensional arrays (could not use 3D arrays) • Solution: Used 1D array with an indexing key to make it act like a 3D array • Challenge: Storing the pixels in an order that can be appropriately displayed on the Perspecta • Solution: Needed to think in 3D and imagine how pixels would be organized in a 3D structure

8. Challenge – Displaying Image • Challenge: Voxels are drawn from the bottom moving up and forward on the Perspecta • Solution: Modify the indexing key to store image pixels so that they are displayed on the Perspecta in appropriate order

9. Challenge - Displaying Image

10. Challenge – Image orientation • Challenge: Image slices represent object taken from different angles • Solution1: Modify the transformation matrix to rotate and move the reconstructed object • Solution2: Create multiple indexing keys to change the orientation of the displayed image • Challenge: Changing from one orientation to another changes the data structure dimensions • Solution: Need to change the data structure dimensions as the object orientation is changed

11. Challenge – Image orientation

12. Challenge – Image orientation

13. Challenge – Image orientation

14. Challenge – Colour Map • Challenge: Differentiating between pixels that belong to the segmentation and those that do not • Solution: Store 2 colours in the colour map and then normalize the pixels to within a certain range depending on whether they are part of the segmentation or not

15. Challenge – Colour Map 60/2 = 30 45/2 + 128 = 150 210/2 + 128 = 233 245/2 = 122

16. Conclusion • The reconstruction was successful!!! • Colouring the segmented portion of the image a different colour worked • Slight loss in sharpness, but barely noticeable • 24 points of orientation achieved • Works but is clumsy • Different data sets are not oriented as expected • Overall my goals were achieved

17. Further Improvements • Extend the colour map to allow for multiple colours • Implement the ability to view colour images • Implement arbitrary angles of rotation • Code may require some modifications to allow for datasets which use non-square images • Ability to add padding to the images • Use actual OpenGL calls to display images on the Perspecta • More options available in OpenGL, quicker learning curve

18. Questions?