target
Understanding the concept of Harris corner detection

* Learning functions: cv2.cornerHarris(), cv2.cornerSubPix()

principle

In the last section, we already know a characteristic of corners: they move in any direction and change a lot. Chris_Harris and Mike_Stephens proposed corner detection method as early as 1988 in A Combined Corner and Edge Detector, which is called Harris corner detection. He converted this simple idea into mathematical form. Move the window in all directions (u, v) and calculate the sum of all the differences. The expression is as follows:

The window function can be either a normal rectangular window or a Gauss window with different weights for each pixel.

In corner detection, E (u; v) should be maximized. This means that the second term on the right side of the equation must be maximized. After Taylor series expansion of the above equation and several steps of mathematical conversion (referring to other standard textbooks), we get the following equation:

Among them:

Here Ix and Iy are the derivatives of the image in the x and y directions. (You can use the function cv2.Sobel().
Then there's the main part. They scored according to an equation that determines whether the window contains corners or not.

Among them:

So according to these characteristics, we can judge whether an area is a corner, boundary or plane.

Our conclusions can be expressed in the following figure:

So the result of Harris corner detection is a gray image composed of corner points. By choosing the appropriate threshold to binarize the result image, we can detect the corners in the image. We'll demonstrate it with a simple picture.

30.1 Harris Corner Detection in OpenCV

Corner Harris (src, blockSize, ksize, K [, dst [, borderType]]) dst can be used for corner detection. The parameters are as follows:

Parameters:

• src – Input single-channel 8-bit or floating-point image.
• dst – Image to store the Harris detector responses. It has the typeCV_32FC1 and the same size assrc .
• blockSize – Neighborhood size (see the details oncornerEigenValsAndVecs() ).
• ksize – Aperture parameter for theSobel() operator.
• k – Harris detector free parameter. See the formula below.
• borderType – Pixel extrapolation method. SeeborderInterpolate() .

img - Input image with data type float32.
blockSize - The size of the area to be considered in corner detection.
The window size used in ksize - Sobel derivation
The free parameters in the k-Harris corner detection equation are [0,04,0.06].
Examples are as follows:

1. <span style="font-size:12px;"># -*- coding: utf-8 -*-  
2. """ 
3. Created on Mon Jan 20 18:53:24 2014 
4. @author: duan 
5. """  
6. import cv2  
7. import numpy as np  
8. filename = 'chessboard.jpg'  
9. img = cv2.imread(filename)  
10. gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)  
11. gray = np.float32(gray)  
12. #The input image must be float32, with the last parameter ranging from 0.04 to 0.05  
13. dst = cv2.cornerHarris(gray,2,3,0.04)  
14. #result is dilated for marking the corners, not important  
15. dst = cv2.dilate(dst,None)  
16. # Threshold for an optimal value, it may vary depending on the image.  
17. img[dst>0.01*dst.max()]=[0,0,255]  
18. cv2.imshow('dst',img)  
19. if cv2.waitKey(0) & 0xff == 27:  
20.     cv2.destroyAllWindows()</span>  

                                   

30.2 Corner Points with Subpixel Accuracy
Sometimes we need the most accurate corner detection. OpenCV provides us with the function cv2.cornerSubPix(), which can provide sub-pixel corner detection. Here is an example. First we need to find the Harris corner, and then pass the center of gravity of the corner to the function for correction. Harris corners are marked with red pixels and green pixels are corrected. In using this function, we define an iteration stopping condition. When the number of iterations is reached or the accuracy condition is satisfied, the iteration will stop. We also need to define
Neighborhood size of line corner search.

1. <span style="font-size:12px;"># -*- coding: utf-8 -*-  
2. """ 
3. Created on Mon Jan 20 18:55:47 2014 
4. @author: duan 
5. """  
6. import cv2  
7. import numpy as np  
8. filename = 'chessboard2.jpg'  
9. img = cv2.imread(filename)  
10. gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)  
11. # find Harris corners  
12. gray = np.float32(gray)  
13. dst = cv2.cornerHarris(gray,2,3,0.04)  
14. dst = cv2.dilate(dst,None)  
15. ret, dst = cv2.threshold(dst,0.01*dst.max(),255,0)  
16. dst = np.uint8(dst)  
17. # find centroids  
18. #connectedComponentsWithStats(InputArray image, OutputArray labels, OutputArray stats,  
19. #OutputArray centroids, int connectivity=8, int ltype=CV_32S)  
20. ret, labels, stats, centroids = cv2.connectedComponentsWithStats(dst)  
21. # define the criteria to stop and refine the corners  
22. criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.001)  
23. #Python: cv2.cornerSubPix(image, corners, winSize, zeroZone, criteria)  
24. #zeroZone – Half of the size of the dead region in the middle of the search zone  
25. #over which the summation in the formula below is not done. It is used sometimes  
26. # to avoid possible singularities of the autocorrelation matrix. The value of (-1,-1)  
27. # indicates that there is no such a size.  
28. #The return value is an array of corner coordinates (not images)  
29. corners = cv2.cornerSubPix(gray,np.float32(centroids),(5,5),(-1,-1),criteria)  
30. # Now draw them  
31. res = np.hstack((centroids,corners))  
32. #np.int0 can be used to omit the number after the decimal point (not four or five entries).  
33. res = np.int0(res)  
34. img[res[:,1],res[:,0]]=[0,0,255]  
35. img[res[:,3],res[:,2]] = [0,255,0]  
36. cv2.imwrite('subpixel5.png',img)</span>  

The results are as follows. For the convenience of viewing, we enlarged the diagonal part.