OpenCV 2.3 Tutorial

1. OpenCV 2.3 Tutorial An Introduction

2. Ouline • Introduction • GUI commands • Primitive data structures • Working with images • Working with videos

3. What is OpenCV • Open source computer vision library in C language. • Generic image/video acquisition and manipulation • Optimized for Intel and intended for real-time applications • Intel Image Processing Library (IPL)

4. OS and Language Supports • Supported OS: • Linux, Mac OS, Windows, and Android. • Language supports: • C/C++, and Python • Third party wrappers • C# (Emgu CV) http://www.emgu.com/wiki/index.php/Main_Page • Java (JavaCV) http://code.google.com/p/javacv/

5. OpenCV Library • opencv_highgui.lib: High-level GUI • opencv_core.lib: Core Library • opencv_imgproc.lib: Image Processing • opencv_video.lib: Video Analysis • opencv_gpu.lib: GPU-accelerated Computer Vision • opencv_ml.lib: Machine Learning • And many more…

6. Supported Sources • Supported image formats: • BMP, DIB, JPEG, JPG, JPE, PNG, PBM, PGM, PPM, SR, RAS, TIFF, TIF • Supported video formats: • AVI file • Web Camera • Live Stream

7. Naming Conventions • Function naming conventions: cvActionTargetMod(...) • E.g.: cvLoadImage, cvReleaseImage • Matrix data types: CV_<bit_depth>(S|U|F)C<number_of_channels> • S = Signed integer • U = Unsigned integer • F = Float • E.g.: CV_8UC1, CV_32FC2

8. Naming Conventions (Cont’d) • Image data types: IPL_DEPTH_<bit_depth>(S|U|F) • E.g.: IPL_DEPTH_8U, IPL_DEPTH_32F • Header files: • #include <cv.h> • #include <cvaux.h> • #include <highgui.h>

9. GUI Commands Window management Input handling

10. Window Management • Load an image: IplImage* img=0; img=cvLoadImage(fileName); if(!img) printf("Could not load image file: %s\n",fileName); • Create and position a window: cvNamedWindow("win1", CV_WINDOW_AUTOSIZE); cvMoveWindow("win1", 100, 100);

11. Window Management • Display an image: cvShowImage("win1",img); • Close a window: cvDestroyWindow("win1"); • Resize a window: cvResizeWindow("win1",100,100);

12. Event Handling • Mouse Events • Keyboard Events • Trackbar Events

13. (1) Mouse Events: 2 Steps • Define a mouse handler: void mouseHandler(int event, int x, int y, int flags, void* param) { switch(event){ case CV_EVENT_LBUTTONDOWN: if(flags & CV_EVENT_FLAG_CTRLKEY) printf("Left button down with CTRL pressed\n"); break; case CV_EVENT_LBUTTONUP: printf("Left button up\n"); break; } } • Register the handler: mouseParam=5; cvSetMouseCallback("win1",mouseHandler, &mouseParam);

14. (2) Keyboard Events • Get keyboard input without blocking: int key; key=cvWaitKey(10); // wait 10ms for input • Get keyboard input with blocking: int key; key=cvWaitKey(0); // wait indefinitely for input • When used in a loop: while(1){ key=cvWaitKey(10); if(key==27) break; switch(key){ case 'h': ... break; case 'i': ... break; } }

15. (3) Trackbar Events: 2 Steps • Define a trackbar handler: void trackbarHandler(int pos) {printf("Trackbar position: %d\n",pos); } • Register the handler: inttrackbarVal=25, maxVal=100; cvCreateTrackbar("bar1", "win1", &trackbarVal ,maxVal , trackbarHandler);

16. Primitive Data Structures CvPoint, CvSize, CvRect, CvScalar CvArr, CvMat, IplImage

17. Primitive Data Structures

18. CvPoint - A Point 0 1 2 3 4 5 6 0 1 2 3 4 CvPoint p = cvPoint(4,2)

19. CvSize – Image Size 0 1 2 3 4 5 6 CvSize r = cvSize(5,7) 0 1 2 3 4

20. CvRect – A Portion of Image 0 1 2 3 4 5 6 CvRect r = cvRect(2,1,4,3) 0 1 2 3 4

21. CvScalar CvScalar |-- double val[4]; //A container of 4-tuple (ch1,ch2,ch3,ch4) of doubles 0 1 2 3 4 5 6 0 1 2 3 4 B G R A CvScalar s = cvScalar(150,57,111,0)

22. Primitive Data Type (Cont’d) CvArr<<Abstract>> CvMat IplImage

23. 2D Matrix CvMat |-- int type; // elements type (uchar,short,int,float,double) and flags |-- int step; // full row length in bytes |-- int rows, cols; // dimensions |-- int height, width; // alternative dimensions reference |-- union data; |-- uchar* ptr; // data pointer for an unsigned char matrix |-- short* s; // data pointer for a short matrix |-- int* i; // data pointer for an integer matrix |-- float* fl; // data pointer for a float matrix |-- double* db; // data pointer for a double matrix

24. ND-Matrix CvMatND |-- int type; // elements type (uchar,short,int,float,double) and flags |-- int dims; // number of array dimensions |-- union data; | |-- uchar* ptr; // data pointer for an unsigned char matrix | |-- short* s; // data pointer for a short matrix | |-- int* i; // data pointer for an integer matrix | |-- float* fl; // data pointer for a float matrix | |-- double*db; // data pointer for a double matrix | |-- struct dim[]; // information for each dimension |-- size; // number of elements in a given dimension |-- step; // distance between elements in a given dimension

25. Things you can do with CvMat • Add • Sub • Div • CrossProduct • DotProduct • Sum • Trace • Transpose • Det • CalcCovarMatrix • EigenVV • InRange • InvSqrt • Invert • IsInf • IsNaN • Mahalonobis • Merge • Solve • SolveCubic • Sqrt • SVD

26. IplImage |-- intnChannels; // Number of color channels (1,2,3,4) |-- intdepth; // Pixel depth in bits: | // IPL_DEPTH_8U, IPL_DEPTH_8S, | // IPL_DEPTH_16U,IPL_DEPTH_16S, | // IPL_DEPTH_32S,IPL_DEPTH_32F, | // IPL_DEPTH_64F |-- intwidth; // image width in pixels |-- intheight; // image height in pixels |-- char* imageData;// pointer to aligned image data. Color images are stored in BGR order |-- intdataOrder; // cvCreateImage can only create interleaved images | // 0 - interleaved color channels, | // 1 - separate color channels |-- intorigin; // 0 - top-left origin, | // 1 - bottom-left origin (Windows bitmaps style, flipped) |-- intwidthStep; // size of aligned image row in bytes |-- intimageSize; // image data size in bytes = height*widthStep |-- struct _IplROI *roi; // image ROI. when not NULL specifies image | // region to be processed. |-- char *imageDataOrigin; // pointer to the unaligned origin of image data | // (needed for correct image deallocation) |-- int align; // Alignment of image rows: 4 or 8 byte alignment | // OpenCV ignores this and uses widthStep instead |-- char colorModel[4]; // Color model - ignored by OpenCV

27. Interleaved Color Channels IplImage : |-- intdataOrder; // cvCreateImage() only creates interleaved images // 0 - interleaved color channels, // 1 - separate color channels 0 1 2 3 4 5 6 0 1 2 3 4 3-channel image (interleaved) [0] [1] [2]

28. Aligned Image Data X X X X … 0 1 2 3 4 5 6 0 1 2 3 4 IplImage : |-- char* imageData; // pointer to aligned image data |-- intwidthStep; // size of aligned image row in bytes |-- intimageSize; // image data size in bytes = height*widthStep 1st row 2nd row imageData imageSize-1 . . . . . . . . . . ... widthStep widthStep

29. Accessing Image Elements IplImage* src=cvLoadImage(“lena.jpg”); uchar *aPixelIn = (uchar *) src->imageData; for ( intiRow = 0; iRow < src->height; iRow-- ) { for ( intiCol = 0; iCol < src->width; iCol++ ) { int R, G, B; B = aPixelIn[iRow*src->widthStep + iCol*src->nChannels + 0 ]; G = aPixelIn[iRow*src->widthStep + iCol*src->nChannels + 1 ]; R = aPixelIn[iRow*src->widthStep + iCol*src->nChannels + 2 ]; } } [0] [1] [2] 1st row 2nd row aPixelIn imageSize-1 . . . . . . . . . . ... widthStep widthStep

30. Working with Images Allocating and releasing images Reading and writing images Image conversion Drawing commands

31. Allocating and releasing images • Reading and writing images • Image conversion & annotation

32. Allocate an Image Usage: IplImage* cvCreateImage(CvSize size, int depth, int channels); Examples: // Allocate a 1-channel byte image IplImage* img1=cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,1); // Allocate a 3-channel float image IplImage* img2=cvCreateImage(cvSize(640,480),IPL_DEPTH_32F,3); Note: • It is developer’s responsibility to de-allocate the memory

33. Clone an Image Usage: IplImage* cvCloneImage(const IplImage* image) Example: IplImage* img1= cvCreateImage(cvSize(640,480), IPL_DEPTH_8U,1); IplImage* img2=0; img2=cvCloneImage(img1); // need to free memory later Note: • It is developer’s responsibility to de-allocate the memory

34. Release an Image Usage: void cvReleaseImage(IplImage** image) Example: IplImage* img= cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,1); cvReleaseImage(&img); Note: • You MUST release memory for cvCreateXX, cvCloneXX by calling cvReleaseXX. Or memory leak occurs.

35. Region of Interest of an Image Usages: void cvSetImageROI(IplImage* image, CvRectrect); void cvResetImageROI(IplImage* image); CvRectcvGetImageROI(const IplImage* image); Example: If (!cvGetImageROI(src)) cvResetImageROI(src); CvRect mask = cvRect(0,0,320,120); cvSetImageROI(src, mask); NOTE: • The majority of OpenCV functions support ROI.

36. Allocating and releasing images • Reading and writing images • Image conversion & annotation

37. Reading an Image Usage: IplImage* cvLoadImage(const char* filename, intiscolor=CV_LOAD_IMAGE_COLOR) Example: IplImage* img=cvLoadImage(“lena.jpg”); if(!img) printf(“Failed to load lena.jpg\n”); Note: • Can be color or grayscale byte/float-image • Byte image: 0-255; Float image: 0..1 • A color image is assumed to have data in BGR order

38. Saving an Image Usage: intcvSaveImage(const char* filename, constCvArr* image) Example: if(!cvSaveImage(“out.jpg”, img)) printf("Could not save out.jpg.\n”); Note: • The output image format is determined based on the file extension.

39. Allocating and releasing images • Reading and writing images • Image conversion & annotation

40. Color Model Conversion Usage: void cvCvtColor(const CvArr* src, CvArr* dst, int code) • Color code: CV_BGR2GRAY, CV_BGR2HSV, CV_BGR2Lab Example: IplImage * dst= cvCreateImage(cvGetSize(src), src->depth, 1); cvCvtColor(src ,dst, CV_RGB2GRAY); Note: • Color spaces: RGB, BGR, GRAY, HSV, YCrCb, XYZ, Lab, Luv, HLS

41. Ex. RGB  Lab R G B RGB Image Lab Image L a b

42. Image Conversion (8-bit16-bit) Usage: void cvConvertScale(const CvArr* src, CvArr* dst, double scale=1, double shift=0) Example: // IplImage *src is a 8-bit unsigned color image IplImage *img32=0; img32= cvCreateImage(cvGetSize(src),IPL_DEPTH_32F, src->nChannels); cvConvertScale(src, img32);

43. Image Conversion (Flip) Usage: void cvConvertImage( const CvArr* src, CvArr* dst, int flags=0 ); • CV_CVTIMG_FLIP - flip the image vertically • CV_CVTIMG_SWAP_RB - swap red and blue channels Example: IplImage *img=0; img= cvCreateImage(cvGetSize(src), src->depth, src->nChannels); cvConvertImage(src, img, CV_CVTIMG_FLIP);

44. Resize an Image Usage: void cvResize(const CvArr*src, CvArr*dst, int interpolation=CV_INTER_LINEAR) Example: // resize to half of original size IplImage *dst = cvCreateImage( cvSize((int)(src->width*0.5),(int)(src->height*0.5)), src->depth, src->nChannels); cvResize(src,dst);

45. Annotate an Image Usage: void cvCircle(CvArr*img, CvPoint center, int radius, CvScalar color, int thickness=1, intlineType=8, int shift=0) void cvLine(CvArr*img, CvPoint pt1, CvPoint pt2, CvScalar color, int thickness=1, intlineType=8, int shift=0) void cvPutText(CvArr* img, const char* text, CvPoint org, const CvFont* font, CvScalar color) Example: cvCircle(img, cvPoint(100,100), 20, cvScalar(0,255,0), 1);

46. Working with Videos Retrieve an image from video Access video information Write a video file

47. Open from a Video • From Camera: CvCapture* capture = cvCaptureFromCAM(0); • From an AVI File: CvCapture* capture = cvCaptureFromAVI("infile.avi"); • From a Live Stream: CvCapture* capture = cvCaptureFromFile (“rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov"); If OpenCVfailed to connect to the source, cvCaptureFromXX(..) returns NULL.

48. Grab an Image // make sure the connection succeeded If (capture ==0) { printf(“Failed to open video source.\n”); return; } IplImage* img = cvQueryFrame( capture ); if (img==0) { printf(“Failed to grab image.\n”); return; } cvShowImage(“myWindow”, img); • Whenever you grab images from CvCapture object, there is NO need to call cvReleaseImage(). • It is CvCapture’s responsibility to free the memory!

49. Release the Capture Source Example cvReleaseCapture(&capture); Note: • Images captured by the device is allocated/released by the capture function. • There is no need to release it explicitly.

50. Access Video Information Usage: double cvGetCaptureProperty(CvCapture* capture, intproperty_id) intcvSetCaptureProperty(CvCapture* capture, intproperty_id) Example: int height = (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT); int width= (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH); int fps = (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FPS); int count = (int) cvGetCaptureProperty(capture, CV_CAP_PROP_FRAME_COUNT);