IM.Grid : A Grid Computing Solution for image processing

1. Image Mining Group @ Institut Pasteur Korea HongKee Moon IM.Grid: A Grid Computing Solution for image processing

2. Introduction • Goals • Increase speed of computation • Make it much faster! • How? • Super powerful computer • Distributed computing • Distributed computing • Multithreading • Thread is a light-weight processing unit • A program can have multiple threads running simultaneously • Grid computing • A program exploits multiple remote computers • Difficult to implement • GPGPU-Grid computing • A program exploits multiple remote computers powered by GPGPU • Very difficult to implement

3. Index • Goals & Structure • Comparisons • Demo • Multithreads & Grids • Results • Possible applications

4. Goals • Reliability • High performance

5. Image Processing Pipeline

6. Structure

7. Structure

8. Batch Processing Load an image Process the image Results Results

9. Multithreading Load two images Process the images Results

10. Grid Computing Grid Computing Network Results Load ten images Process the images

11. GPGPU-Grid Computing GPGPU-Grid Computing Network Results Load ten images Process the images

12. IM 1.0 • IM 1.0 • Batch process mode • Process screening well by well • Takes 384 seconds (if it takes 1 second for processing a well of 384 well-plate) • Costs less memory but slow

13. IM 2.0 + Multithreads • IM 2.0 + Multithreads • Multithreaded process • Process screening multiple wells simultaneously • Takes 384/N seconds (N: number of threads) • High performance • Needs large amounts of memory for loading N images • Out-of-memory exception

14. IM 2.0 + Grids • IM 2.0 + Grids • Same as Multithreaded process • Process screening multiple wells simultaneously • Takes 384/M seconds (M: number of grids) • Communication to grids • Guarantees high performance with less memory • Suitable for HT-HCS

15. IM 2.0 + GPGPU + Grids

16. Demo • Compare batch-mode and multithreads • IM2 multithread demo.avi • Compare multithreads and grid computing • Multithreads Video.avi • Dell 690 workstation 2 1.8GHz Quad-core with 4GB memory • Grid Video.avi • 10 Dell PowerEdge Blade Servers • Each grid has 2 3GHz processors with hyper-threading and 2GB memory

17. Multithreads & Grids • Multithreads • Multiple instances of plugin • Allocates (N x plugin instance) + (N x images) in the memory of one computer • Image data is loaded in PC  high network bandwidth • Better performance if PC has multi cores • Grids • Multithreads only for communication with multiple computers  sends parameters and receives results • Less memory usage • Image data is loaded in each grid  low network bandwidth • Performance is guaranteed regardless of PC performance

18. Result 1

19. Result 2

20. Possible applications • High Throughput–High Content Screening • Search for optimal parameters during new algorithm development • Solve complex problems with divide-conquer strategy • Can be used for another general purpose • Let us know if you want to use Grids!

21. Reference • “IM.Grid, a Grid Computing Approach in High Throughput-High Content Screening”, The 9th IEEE/ACM International Conference on Grid Computing, 2008