1 / 61

OpenCV Tutorial

OpenCV Tutorial. Omri Perez Adapted from: Gary Bradski Senior Scientist, Willow Garage Consulting Professor: Stanford CS Dept. http://opencv.willowgarage.com www.willowgarage.com. 1. Vision is Hard Camera Model, Lens, Problems and Corrections OpenCV OpenCV Tour. Vision is Hard.

lainey
Download Presentation

OpenCV Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. OpenCV Tutorial Omri Perez Adapted from: Gary Bradski Senior Scientist, Willow Garage Consulting Professor: Stanford CS Dept. http://opencv.willowgarage.com www.willowgarage.com 1

  2. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

  3. Vision is Hard • What is it? • Turning sensor readings into perception. • Why is it hard? • It’s just numbers. Maybe try gradients to find edges? CS324

  4. Use Edges? … It’s not so simple • Depth discontinuity • Surface orientation discontinuity • Reflectance discontinuity (i.e., change in surface material properties) • Illumination discontinuity (e.g., shadow) Slide credit: Christopher Rasmussen CS324

  5. Must deal with Lighting Changes … CS324

  6. Lighting is also a Strong Cue Gary Bradski (c) 2008 6

  7. The Brain Assumes 3D Geometry Perception is ambiguous … depending on your point of view! 7

  8. Geometrical aberrations Non-Geometrical aberrations • spherical distortion • astigmatism • tangential distortion • coma • Chromatic • Vignetting These are typically what are corrected for in camera Calibration aberrations are reduced by combining lenses Marc Pollefeys

  9. Distortion Correction so that Lenscan Approximate a Pinhole Camera • Distortions are corrected mathematically • We use a calibration pattern • We find where the points ended up • We know where the points hould be • OpenCV 2.2 Function: double calibrateCamera( const vector<vector<Point3f> >& objectPoints, const vector<vector<Point2f> >& imagePoints, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs, vector<Mat>& rvecs, vector<Mat>& tvecs, int flags=0); CS324

  10. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

  11. OpenCV Overview: Robot support > 2000 algorithms opencv.willowgarage.com Image Pyramids General Image Processing Functions Geometric descriptors Camera calibration, Stereo, 3D Segmentation Features Utilities and Data Structures Transforms Tracking Machine Learning: • Detection, • Recognition Fitting Matrix Math Gary Bradski

  12. OpenCV Tends Towards Real Time http://opencv.willowgarage.com

  13. Where is OpenCV Used? • Google Maps, Google street view, Google Earth, Books • Academic and Industry Research • Safety monitoring (Dam sites, mines, swimming pools) • Security systems • Image retrieval • Video search • Structure from motion in movies • Machine vision factory production inspection systems • Robotics • Well over 2M downloads 2M downloads Screen shots by Gary Bradski, 2005

  14. OpenCV Modules • Calib3d • Calibration, stereo, homography, rectify, projection, solvePNP • Contrib • Octree, self-similar feature, sparse L-M, bundle adj, chamfer match • Core • Data structures, access, matrix ops, basic image operations • features2D • Feature detectors, descriptors and matchers in one architecture • Flann (Fast library for approximate nearest neighbors) • Gpu – CUDA speedups • Highgui • Gui to read, write, draw, print and interact with images • Imgproc – image processing functions • Ml – statistical machine learning, boosting, clustering • Objdetect – PASCAL VOC latent SVM and data reading • Traincascade – boosted rejection cascade CS324

  15. Software Engineering • Works on: • Linux, Windows, Mac OS (+ Android since open CV 2.2) • Languages: • C++, Python, C • Online documentation: • Online reference manuals: C++, C and Python.

  16. Vision is Hard • Camera Model, Lens, Problems and Corrections • OpenCV • OpenCV Tour CS324

  17. Gradients: Scharr instead of Sobel • Sobel has been the traditional 3x3 gradient finder. • Use the 3x3 Scharr operator instead since it is just as fast but has more accurate response on diagonals. void Scharr(const Mat& src, Mat& dst, int ddepth, int xorder, int yorder, double scale=1, double delta=0, int borderType=BORDER_DEFAULT) CS324

  18. Canny Edge Detector Canny() OpenCV team, Gary Bradski 18

  19. Hough Transform HoughCircles(), HoughLines(), HoughLinesP() (probabilistic Hough) Gary Bradski (c) 2008 19

  20. Scale Space void cvPyrUp( IplImage* src, IplImage* dst, IplFilter filter = IPL_GAUSSIAN_5x5); void cvPyrDown( IplImage* src, IplImage* dst, IplFilter filter = IPL_GAUSSIAN_5x5); Gary Bradski (c) 2008 20

  21. Space Variant vision: Log-Polar Transform cvLogPolar(src,dst,center,size, CV_INTER_LINEAR+CV_WARP_FILL_OUTLIERS) Gary Bradski (c) 2008 21

  22. Delaunay Triangulation, Voronoi Tessellation CvSubdiv2D* cvCreateSubdivDelaunay2D(CvRect rect, CvMemStorage* storage) Gary Bradski (c) 2008 22

  23. Contours void findContours() Gary Bradski (c) 2008 23

  24. Histogram Equalization void equalizeHist(const Mat& src, Mat& dst) Gary Bradski (c) 2008 24

  25. Image textures • Inpainting: • Removes damage to images, in this case, it removes the text. void inpaint(const Mat& src, const Mat& inpaintMask, Mat& dst, double inpaintRadius, int flags); Gary Bradski (c) 2008 25

  26. Morphological Operations Examples • Morphology - applying Min-Max. Filters and its combinations Void morphologyEx() createMorphologyFilter() erode() dilate() Dilatation IB Image I Erosion IB Opening IoB= (IB)B Grad(I)= (IB)-(IB)‏ Closing I•B= (IB)B TopHat(I)= I - (IB)‏ BlackHat(I)= (IB) - I Gary Bradski (c) 2008 26

  27. Distance Transform • Distance field from edges of objects void distanceTransform(const Mat& src, Mat& dst, int distanceType, int maskSize) Flood Filling int floodFill(Mat& image, Point seed, Scalar newVal, Rect* rect=0, Scalar loDiff=Scalar(), Scalar upDiff=Scalar(), int flags=4) 27 Gary Bradski (c) 2008

  28. Thresholds void adaptiveThreshold() double threshold() Gary Bradski (c) 2008 28

  29. Segmentation • Pyramid, mean-shift, graph-cut • Here: Watershed void watershed(const Mat& image, Mat& markers) Gary Bradski (c) 2008 29

  30. Background Subtraction BackgroundSubtractorMOG2(), see samples/cpp/bgfg_segm.cpp Gary Bradski (c) 2008 30

  31. Image Segmentation & Minimum Cut Pixel Neighborhood w Image Pixels Similarity Measure Minimum Cut * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

  32. GrabCut void grabCut(const Mat& image, Mat& mask, Rect rect, Mat& bgdModel, Mat& fgdModel, int iterCount, int mode) • Graph Cut based segmentation Gary Bradski

  33. Motion Templates (My work with James Davies)‏ • Object silhouette • Motion history images • Motion history gradients • Motion segmentation algorithm silhouette MHI MHG Gary Bradski (c) 2008 33

  34. Segmentation, Motion Tracking void updateMotionHistory(); void calcMotionGradient(); double calcGlobalOrientation(); Motion Segmentation Motion Segmentation Pose Recognition Gesture Recognition 34 Gary Bradski (c) 2008 James Davies, Gary Bradski

  35. Tracking with CAMSHIFT • Control game with head RotatedRect CamShift(const Mat& probImage, Rect& window, TermCriteria criteria)

  36. 3D tracking • Camera Calibration • View Morphing • POSIT • void POSIT() • A more general technique for solving pose is • solving the Percpective N Point problem: • void solvePnP(…)

  37. Mean-Shift for Tracking CamShift(); MeanShift(); Gary Bradski (c) 2008 37

  38. Optical Flow // opencv/samples/c/lkdemo.c int main(…){ … CvCapture* capture = <…> ? cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path); if( !capture ) return -1; for(;;) { IplImage* frame=cvQueryFrame(capture); if(!frame) break; // … copy and process image cvCalcOpticalFlowPyrLK( …)‏ cvShowImage( “LkDemo”, result ); c=cvWaitKey(30); // run at ~20-30fps speed if(c >= 0) { // process key }} cvReleaseCapture(&capture);} calcOpticalFlowPyrLK() Also see dense optical flow: calcOpticalFlowFarneback()

  39. Features 2D Read two input images: Mat img1 = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE); Detect keypoints in both images: // detecting keypoints FastFeatureDetector detector(15); vector<KeyPoint> keypoints1; detector.detect(img1, keypoints1); Compute descriptors for each of the keypoints: // computing descriptors SurfDescriptorExtractor extractor; Mat descriptors1; extractor.compute(img1, keypoints1, descriptors1); Now, find the closest matches between descriptors from the first image to the second: // matching descriptors BruteForceMatcher<L2<float> > matcher; vector<DMatch> matches; matcher.match(descriptors1, descriptors2, matches); CS324

  40. Features 2D continued … Viusalize the results namedWindow("matches", 1); Mat img_matches; drawMatches(img1, keypoints1, img2, keypoints2, matches, img_matches); imshow("matches", img_matches); waitKey(0); Find the homography transformation between two sets of points: vector<Point2f> points1, points2; // fill the arrays with the points .... Mat H = findHomography(Mat(points1), Mat(points2), CV_RANSAC, ransacReprojThreshold); Create a set of inlier matches and draw them. Use perspectiveTransform function to map points with homography: Mat points1Projected; perspectiveTransform(Mat(points1), points1Projected, H); Use drawMatches() again for drawing inliers. CS324

  41. Features2d contents Detection: Description: Detectors available Descriptors available SIFT SURF One way Calonder (under construction) FERNS • SIFT • SURF • FAST • STAR • MSER • GFTT (Good Features To Track)

  42. Kalman Filter, Partical Filter for Tracking Kalman Condensation or Particle Filter ::KalmanFilter class ConDensation 42 Gary Bradski (c) 2008

  43. Projections Find: Warp: Mat getAffineTransform() void warpAffine() Mat getPerspectiveTransform() void warpPerspective()

  44. Homography • Maps one plane to another • In our case: A plane in the world to the camera plane • Great notes on this: Robert Collins CSE486 • http://www.cse.psu.edu/~rcollins/CSE486/lecture16.pdf • Derivation details: Learning OpenCV 384-387 Perspective Matrix Equation (camera coords Pt in world to pt on image) Gary Bradski and Adrian Kaehler: Learning OpenCV Gary Bradski, CS223A, Into to Robotics

  45. Homography • We often use the chessboard detector to find 4 non-colinear points • (X,Y * 4 = 8 constraints) • To solve for the 8 homography parmeters. • Code: Once again, OpenCV makes this easy • findHomography(…) or: • getPerspectiveTransform(…) Gary Bradski, CS223A, Into to Robotics

  46. Single Camera Calibration See samples/cpp/calibration.cpp Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds. And after that you’ll get: 3D view of checkerboard Un-distorted image Gary Bradski (c) 2008 46

  47. Stereo … Depth from Triangulation • Involved topic, here we will just skim the basic geometry. • Imagine two perfectly aligned image planes: Depth “Z” and disparity “d” are inversly related:

  48. Stereo • In aligned stereo, depth is from similar triangles: • Problem: Cameras are almost impossible to align • Solution: Mathematically align them: All: Gary Bradski and Adrian Kaehler: Learning OpenCV

  49. Stereo Rectification • Algorithm steps are shown at right: • Goal: • Each row of the image contains the same world points • “Epipolar constraint” Result: Epipolar alignment of features: All: Gary Bradski and Adrian Kaehler: Learning OpenCV

  50. samples/c In ...\opencv_incomp\samples\c bgfg_codebook.cpp - Use of a image value codebook for background detection for collecting objects bgfg_segm.cpp - Use of a background learning engine blobtrack.cpp - Engine for blob tracking in images calibration.cpp - Camera Calibration camshiftdemo.c - Use of meanshift in simple color tracking contours.c - Demonstrates how to compute and use object contours convert_cascade.c - Change the window size in a recognition cascade convexhull.c - Find the convex hull of an object delaunay.c - Triangulate a 2D point cloud demhist.c - Show how to use histograms for recognition dft.c - Discrete fourier transform distrans.c - distance map from edges in an image drawing.c - Various drawing functions edge.c - Edge detection facedetect.c - Face detection by classifier cascade ffilldemo.c - Flood filling demo find_obj.cpp - Demo use of SURF features fitellipse.c - Robust elipse fitting houghlines.c - Line detection image.cpp - Shows use of new image class, CvImage(); inpaint.cpp - Texture infill to repair imagery kalman.c - Kalman filter for trackign kmeans.c - K-Means laplace.c - Convolve image with laplacian. letter_recog.cpp - Example of using machine learning Boosting, Backpropagation (MLP) and Random forests lkdemo.c - Lukas-Canada optical flow minarea.c - For a cloud of points in 2D, find min bounding box and circle. Shows use of Cv_SEQ morphology.c - Demonstrates Erode, Dilate, Open, Close motempl.c - Demonstrates motion templates (orthogonal optical flow given silhouettes) mushroom.cpp - Demonstrates use of decision trees (CART) for recognition pyramid_segmentation.c - Color segmentation in pyramid squares.c - Uses contour processing to find squares in an image stereo_calib.cpp - Stereo calibration, recognition and disparity map computation watershed.cpp - Watershed transform demo.

More Related