Software Resources

1. Software Resources Multimodal Interaction Dr. Mike Spann m.spann@bham.ac.uk http://www.eee.bham.ac.uk/spannm

2. Contents • Introduction • ImageJ introduction • Image representation in ImageJ • Displaying images in ImageJ • ImageJ utility class • Regions of interest • Writing plug-ins • OpenCV introduction • Image handling and display in OpenCV • Other libraries • Key tips • Key resources

3. Introduction • Two free platforms for developing image processing software are ImageJ and OpenCV • ImageJ– Java • OpenCV – C++, C# • They both allow rapid development of image processing algorithms • ImageJ through the use of plugins • Both have extensive libraries of well known image processing functions for the user to build upon

4. ImageJ introduction • All of this material is taken from the ImageJ tutorial accessed from my web-site • http://mtd.fh-hagenberg.at/depot/imaging/imagej/ • Also check out the ImageJ home page • http://rsb.info.nih.gov/ij/ • ImageJ is a free image processing system allowing easy development of Java based image processing algorithms in the form of plug-ins

5. ImageJ introduction • It comes with a user-friendly GUI and can run either as an application or an applet • http://rsb.info.nih.gov/ij/applet/

6. ImageJ introduction • It can handle 8,16,24 and 32 bit images • It can handle most standard image formats • TIFF • GIF • JPEG • BMP • DICOM • It can handle stacks of images • Also there are plug-ins allowing it to handle movie files • It can also handle regions of interest (ROI’s)

7. ImageJ introduction • The key point about ImageJ is that it is simple to add your own algorithms (written as plug-ins) callable from the front-end GUI • File I/O taken care of by ImageJ • Pixel access easy from image handles defined within ImageJ

8. Image representation in ImageJ • ImageJ has 5 built-in image classes • 8 bit grayscale (byte) • 8 bit colour (byte) • 16 bit grayscale (short) • RGB colour (int) • 32 bit image (float) • It also supports image stacks consisting of images (slices) of the same size

9. Image representation in ImageJ • ImageJ uses 2 classes to represent and manipulate images • ImagePlus • An image is represented by an ImagePlus object • ImageProcessor • This holds the pixel data and contains methods to access the pixel data

10. Image representation in ImageJ • Pixel access methods in ImageProcessor include • Object getPixels() – returns a reference to the pixel array (need to cast to appropriate type) • int getHeight() – returns height of pixel array • int getWidth() – returns width of pixel array

11. Image representation in ImageJ • A subclass of ImageProcessor is passed to the run() method of the plug-in filter (see later) • ByteProcessor - 8 bit grayscale images • ShortProcessor – 16 bit grayscale images • ColorProcessor – RGB images • FloatProcessor – 32 bit floating point images

12. Image representation in ImageJ • Pixel representation in ImageJ uses the byte datatype for grayscale and colour images and short for 16-bit grayscale • byte/short are signed data types • byte ranges from –128 to 127 • short ranges from –32768 to 32767 • Obviously grayscale values are usually positive values

13. Image representation in ImageJ • To cast a byte to an integer, we need to eliminate the sign bit • Can cast back the other way easily enough byte[] pixels=(byte[]) ip.getPixels(); int grey=0xxff & pixels[j]; pixels[j]=(byte) grey;

14. Image representation in ImageJ • The ColorProcessor return the pixels as int[] and the RGB values are packed into the one int variable 31 0 bit int[] pixels=(int[]) ip.getPixels(); int red=(0xxff0000 & pixels[j]) >> 16; int green=(0xx00ff00 & pixels[j]) >> 8; int blue=(0xx0000ff & pixels[j]);

15. Image representation in ImageJ • Can reconstitute an RGB array by shifting the other way : pixels[j]=((red & 0xff)<<16)+((green & 0xff)<<8)+(blue & 0xff);

16. Displaying images in ImageJ • A class ImageWindow is used to display ImagePlus objects • We don’t normally need to access methods of ImageWindow • These are automatically called from ImagePlus methods show(), draw() and updateAndDraw()

17. ImageJ utility class • ImageJ contains a class called IJ which contains a number of useful static methods • Error messages • static void error(String message) – displays an error message in a dialog box • static boolean showMessageWithCancel(String title, String message) – allows the user to cancel the plug in or continue

18. ImageJ utility class • Displaying text • static void write(String s) - Outputs text in a window – useful for displaying textual or numerical results of algorithms • Displaying text in a status bar • static void showStatus(String s)

19. ImageJ utility class • User input • static double getNumber(String prompt, double default) – Allows the user to input a number in a dialog box • static String getString(String prompt, String default) – Allows the user to input a string in a dialog box • The GenericDialog class is a more sophisticated way of inputting more than a single number or string

20. Regions of interest (ROI’s) • A plug in filter does not always have to work on the whole image • ImageJ supports ROI’s which can are usually rectangular but can be other shapes as well. • We set/get the ROI using the following method of the ImageProcessor class • void setROI(int x, int y, int w int h) • Rectangle getROI()

21. Writing plug-ins • To write a plug in requires developing a class which implements either the PlugIn or PlugInFilter interfaces • The second is more usual as this is used when the filter requires an input image import ij.*; import ij.plugin.filter.PlugInFilter; import ij.process; class myPlugin implements PlugInFilter

22. Writing plug-ins • Methods setup() and run() must then be provided • Method setup() sets up the plugin filter for use • String arg allows input arguments to be passed to the plugin • Argument imp handled automatically by ImageJ • It’s the currently active image int setup(String arg, ImagePlus imp)

23. Writing plug-ins • Method setup() returns a flag word representing the capabilities of the plug-in filter (for example the types of images it can handle). For example : • static int DOES_8G • static int DOES_RGB • static int DOES_ALL • static int NO_CHANGES (plug in doesn’t change the image data) • static int ROI_REQUIRED • etc

24. Writing plug-ins • The run() method is called when the plugin is run from the ImageJ menu • It contains the code to process individual pixels • If no input image is required • If an input image is required void run(String arg) void run(ImageProcessor ip)

25. Writing plug-ins • Once the Java program is written and compiled, the .class file is stored in the plug-in directory which is a sub-directory of the ImageJ home directory • The class name should contain an underscore symbol (eg MyAlgorithm_ ) • It then appears under the plug-in sub-menu of the ImageJ gui

26. OpenCV introduction • Based on Intel IP Library • More than 300 functions • Features • Multi-platform • Provides a simple window manager with sliders mouse call backs etc • Uses Intel Integrated Performance Primitives to enhance performance • Tutorials & Documentation • Sample code • Download from • sourceforge.net/projects/opencvlibrary/ • Operates under • Windows 95/98/NT/2000/XP • POSIX Linux/BSD/OSX/Win2K/WinXP

27. Image handling and display in OpenCV • Still images from file • Using cvLoadImage() IplImage* cvLoadImage(const char* filename,int iscolor=1); Returns a pointer to an ipl image Filename: the name of the image file iscolor: >0 image is loaded as 3 channel colour 0 image is loaded as grey-level <0 No. of channels determined by file • File formats • Windows bitmaps: bmp, DIB • JPEG files: jpeg, jpg, jpe • Portable Network Graphics: png • Portable image format: pbm, pgm, ppm • Sun raster: sr, ras • Tiff Files: tiff, tif

28. Image handling and display in OpenCV • Simple code to load and display an image • Uses library HighGUI • Need to register image window and display it • int cvNamedWindow(const char* name, int flags);name: id name and text on image bannerint: 1, autosize to fit the image • void cvShowImage(const char* name, const CvArr* img);name: id name and text on image bannerimg: pointer to image to be displayed • CvCapture* cvCaptureFromFile(const char* fname);fname: video file name • CvCapture* cvCaptureFromCAM(int index);index: positive integer specifies video source

29. /* usage: prog <image_name> */ #include "cv.h“ /* OpenCV data types and prototypes */ #include "highgui.h“ /* Image read and display*/ int main( int argc, char** argv ) {IplImage* img; if(argc==2 && (img=cvLoadImage(argv[1], 1)) !=0) {cvNamedWindow("Image view", 1); cvShowImage("Image view", img); cvWaitKey(0); /*very important, contains event */ /*processing loop inside */ cvDestroyWindow("Image view"); cvReleaseImage(&img); return 0; } return -1; }

30. Other libraries:(CV) • Image Processing • Edges, filters, morphologic operators, pyramids, histograms, boundary tracing • Structural analysis • Contour processing, computational geometry • Motion Analysis and Object Tracking • Object tracking, optical flow, motion templates • Pattern Recognition • Camera calibration and 3D Reconstruction • Camera calibration, pose estimation, epipolar geometry

31. Other libraries:cxcore • Operations on arrays • Linear algebra, statistics, random number generator • Dynamic structures • Trees, graphs, sets, sequences • Drawing • Curves, text, point sets, contours

32. Other libraries:ML • Machine Learning • Bayes classifier • K-nearest neighbour classifier • Support Vector Machine • Decision Trees • Boosting • Random Forest • Expectation Maximisation • Neural Networks

33. Other libraries:CvCam/Cvaux • CvCam • Video Input / Output • Select and set up a camera • Render a video stream • Control the Video • Process video frames • Multiple cameras under Linux and Windows • Cvaux • Experimental functions

34. Key tips • IplImage data Types • IPL_DEPTH_8U unsigned 8 bit • IPL_DEPTH_8S signed 8 bit • IPL_DEPTH_16U unsigned 16 bit • IPL_DEPTH_16S signed 16 bit • IPL_DEPTH_32S signed 32 bit • IPL_DEPTH_32F single precision floating point • IPL_DEPTH_ 64F double precision floating point • Channels: 1, 2, 3 or 4 sequence is: b0, g0, r0, b1, g1, r1

35. Key tips • Accessing image pixels • Coordinates are indexed from 0 • Origin is top left or bottom left • To access an element of an 8-bit 1-channel image,I (IplImage* img): I(x,y)~ ((uchar*)(img->imageData + img->widthStep*y)[x]

36. Key tips • To access an element of an 8-bit 3-channel image,I (IplImage* img):I(x,y)blue~ ((uchar*)(img->imageData + img->widthStep*y)[x*3]I(x,y)green~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+1]I(x,y)red~ ((uchar*)(img->imageData + img->widthStep*y)[x*3+2] • To access an element of a 32-bit floating point 1-channel image, I (IplImage* img):I(x,y)~ ((float*)(img->imageData + img->widthStep*y)[x]

37. Key tips • In general to access an element of an N-channel image of type TI(x,y)c~ ((T*)(img->imageData + img->widthStep*y)[x*N+c] • More simply using the provided macroCV_IMAGE_ELEM(img, T, y, x*N+c) • Efficient way to increment brightness of point (100, 100) by 30CvPoint pt = {100, 100};uchar *tmp_ptr=&((uchar*)(img->ImageData + img->widthStep*pt.y))[x*3];tmp_ptr[0]+=30;tmp_ptr[1]+=30;tmp_ptr[2]+=30;

38. Key resources • A brief intro to OpenCV:www.cs.iit.edu/~agam/cs512/lect-notes/opencv-intro/index.html • A more detailed guide (also describes DirectX/DirectShow):http://www.site.uottawa.ca/~laganier/tutorial/ opencv+directshow/cvision.htm • Wikipediahttp://en.wikipedia.org/wiki/OpenCV • BookLearning OpenCV: Computer Vision with the OpenCV Library (Paperback) Gary Bradski and Adrian KaehlerO'Reilly Media, Inc. (15 Jul 2008) ISBN 0596516134