Recently, I studied opencv under the guidance of my teacher. I checked it on the Internet, copied it everywhere, and then changed it. Finally, I finished the first task assigned by the teacher - separating all the characters in all the pictures in the two folders. The picture is as follows:
The upper one is inclined 5 degrees, the middle one is inclined 7 degrees, and the lower one is cut successfully. A horizontal projection and a vertical projection are OK. I won't go into details here. It's hard to make a 7-degree projection. When it's vertical projection, sometimes there will be overlaps, which can't get effective blank space.
At first, I was going to rotate the overlapped pictures to get the gap between them, and then use the projection method, but after the rotation, the positioning became complicated. So I thought if I could get a certain angle through projection, but I found that opencv has no relevant API, so I can't write it by myself. In essence, projection is just to get the sum of pixel values on a line. Here is my oblique projection function:
void slantreduce(Mat src, Mat& hist_ver, double angle) { //The matrix of Mat type whose hist never is 1*src.col is used to store the obtained pixels and //src is the original image and angle is the angle int n; //Single pixel value reading int sum = 0;//accumulator int x, y; //Pixel position (x,y) Mat src1; src.convertTo(src1, CV_32S);//In this case, if the CV is 32S, the pixel cannot be read //n = src1.at<int>(0, 0); //Cout < < n; / / check if there is any error for (int i = 0;i < src1.cols;i++)//The point at which the tilt begins { for (y = 0;y < src1.rows;y++) { x = i - y / tan(angle*PI / 180);//Conversion of relations if (x > 0) { n = src1.at<int>(Point(x, y)); sum = sum + n; //cout << n; } else break; } hist_ver.at < int >(0, i) = sum; sum = 0; } //cout << hist_ver<<endl; }
ok, let's start to summarize my thoughts and take stock of some pits I have encountered during this period.
The first and most important step, image preprocessing, image preprocessing directly determines the difficulty of post-processing image. Unfortunately, I fell into a hole in the first step. After I finished writing the whole program, the success rate at that time was not enough. After the teacher looked at my effect, he said that you have reversed the order of binarization and filtering. At present, the correct process of image processing should be
Binarization, filtering (denoising), corrosion expansion
Upper function
Mat predo(Mat src) { Mat src_blur, img; cvtColor(src, src, COLOR_BGR2GRAY);//Convert to grayscale //Adaptive threshold (SRC? Blur, IMG, 255, CV? Adaptive? Threshold? Mean? C, threshold? Binary, 5, 1); / / adaptive threshold threshold(src, img, 245, 255, CV_THRESH_BINARY); //imshow("preimg1", img); medianBlur(img, img, 5);//median filtering Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(1, 5)); morphologyEx(img, img, MORPH_OPEN, kernel);//Open operation Mat structureElement = getStructuringElement(MORPH_ELLIPSE, Size(3, 1), Point(-1, -1)); //dilate(src, dst, structureElement, Point(-1, -1), 1);// img = 255 - img; ; return img; }
At first, I used adaptive threshold, which was very good. Later, I found that it was not suitable for my image without brightness and contrast, and the effect was not as good as the custom threshold.
The second step is to divide the picture into four small pictures for easy processing. My problem here is to cut horizontally first, and then find that it is better to cut vertically first, because the vertical interval is relatively large and convenient.
There are also some pictures that are not so positive. They need to be corrected by affine rotation before processing.
Normal conditions are as follows:
Abnormal pictures are as follows:
Figure after abnormal handling
When dealing with abnormal pictures, it will be found that it is better to cut vertically first, but not horizontally. The first picture has nothing to say. It's two projections. The second picture is also the most serious place for me to fall into the pit. I searched for the relevant API on the Internet for a long time and couldn't realize the rectification (I'm too busy). Finally, I wrote a function to find the angle and then affine rotation.
Obtain the pixel positions of the 2 and 8 heads (find the first pixel not 0 from the top to the bottom, in order to judge whether it is noisy or not, increase the number to judge whether the next three grids are 0 or not), then the angle α that the picture needs to be tilted can be obtained. (however, due to bug s in later programs, I get the following pixel positions of 2 and 8)
Upper function
void findangle(Mat img, double& angle) { int Numc = img.cols; int Numr = img.rows; int zlen = 18; //2 approximate length of characters int n; //Take out 28 parts separately to reduce workload Mat before; Mat after; Mat hist_ver1, hist_ver2; before = img(Rect(0, 0, zlen, Numr)); after = img(Rect(Numc - zlen, 0, zlen, Numr)); reduce(before / 255, hist_ver1, 1, CV_REDUCE_SUM, CV_32S); reduce(after / 255, hist_ver2, 1, CV_REDUCE_SUM, CV_32S); transpose(hist_ver1, hist_ver1); transpose(hist_ver2, hist_ver2); double x[2] = { 0 }, y[2] = { 0 }; //Store the location of the top pixel for (int i = 0;i < 2;i++) { for (int j = Numr - 1;j >= 0;j--) { if (i == 0) n = hist_ver1.at<int>(0, j); else n = hist_ver2.at<int>(0, j); if (n != 0 && j > 3) { for (int k = j;k > j - 3;k--) { if (i == 0) n = hist_ver1.at<int>(0, k); else n = hist_ver2.at<int>(0, k); if (n == 0) { j = k; break; } if (k == j - 2) { y[i] = j; } } } if (y[i] != 0) break; } } int startn; img.convertTo(img, CV_32S); for (int i = 0;i < 2;i++) { if (i == 0) startn = 0; else startn = Numc - zlen; for (int j = startn;j < zlen + startn;j++) { n = img.at<int>(y[i], j); if (n != 0) { x[i] = j; break; } } } angle = atan(((y[1] - y[0]) / (x[1] - x[0]))) * 180 / PI; }
Simple image segmentation
void partimg(Mat src, vector<Rect>& r1) { Mat hist_ver; reduce(src / 255, hist_ver, 1, CV_REDUCE_SUM, CV_32S); //The image after preprocessing is first projected in abscissa, and the sum of all lines is taken. A value of 0 means that the line may be an interval transpose(hist_ver, hist_ver); //Column switching int count = 0; int width = 10;//Greater than this interval is an area int totaln = hist_ver.cols; //Number of columns for image int reflen = 0; Mat locations = Mat::zeros(3, totaln, CV_32S);//3 * column space //imshow("sec", src); Find_begin_end_len(hist_ver, locations, count, width, 5);//This function is below // cout << hist_ver << endl; // cout << count<<endl; for (int i = 0;i < count;i++) { int x = 0; int y = locations.at<int>(0, i); int bwid = src.cols; int height = locations.at<int>(2, i); int count2 = 0; Rect r2(x, y, bwid, height); Mat hist_vor; reduce(src(r2) / 255, hist_vor, 0, CV_REDUCE_SUM, CV_32S); int blankwidth = 21; Mat locations2 = Mat::zeros(3, totaln, CV_32S);//3 * column space if (i == 0) { Find_begin_end_lencol(hist_vor, locations2, count2, blankwidth, reflen, 8);//This function is below reflen = locations2.at<int>(2, 0); if (count2 > 2) count2 = 2; //cout << count2 << endl; } else { Find_begin_end_lencol(hist_vor, locations2, count2, blankwidth, reflen, 8); if (count2 > 2) count2 = 2; } for (int j = 0;j < count2;j++) { Rect R1(locations2.at<int>(0, j), locations.at<int>(0, i), locations2.at<int>(2, j), locations.at<int>(2, i)); //imshow(to_string(j + i), src(R1)); //cout << R1 << endl; r1.push_back(R1); } } /*r1.push_back(Rect(0, 0, 50, 50));*/ }
This is a vertical projection and a horizontal projection to find the starting and ending positions of the characters
void Find_begin_end_len(Mat hist_ver, Mat& locations, int& count, int width, int minthre) { int N = hist_ver.cols; int begin, end, len; int n = 0; // number of pixels in h_vec[i] //cout << hist_ver << endl; for (int i = 0; i < N; i++) { n = hist_ver.at<int>(0, i); if (n > minthre) { begin = i; for (int j = i; j < N; j++) { n = hist_ver.at<int>(0, j); ///Eliminate misjudgment/// if (n < minthre && j + 2 < N) { for (int k = j;k < j + 2;k++) { if (hist_ver.at<int>(0, k) > 20) { n = n + 20; } } } if (n < minthre) { end = j - 1; len = end - begin; if (len >= width) { locations.at<int>(0, count) = begin; locations.at<int>(1, count) = end; locations.at<int>(2, count) = len; count = count + 1; } i = j; break; } } } } } void Find_begin_end_lencol(Mat hist_ver, Mat& locations, int& count, int blankwidth, int reflen, int minn) { int N = hist_ver.cols; int begin, end, len = 0, zlen; int n = 0; // number of pixels in h_vec[i] //cout <<hist_ver<<endl; for (int i = 0; i < N; i++) { n = hist_ver.at<int>(0, i); if (n > 2) { begin = i; for (int j = i; j < N; j++) { n = hist_ver.at<int>(0, j); if (n < minn && j + 2 < N && j > 2) { if (hist_ver.at<int>(0, j - 1) < minn && hist_ver.at<int>(0, j + 1) < minn) { len++; if (len > blankwidth && hist_ver.at<int>(0, j + 2) >= 3 || j == N - 3) { end = j - len; zlen = end - begin; if (zlen < reflen*0.8&&count == 0) { //cout << reflen; continue; } ////cout << reflen<<endl; //cout << begin<<" "<<end<<" " << zlen<<endl; locations.at<int>(2, count) = zlen; locations.at<int>(0, count) = begin; locations.at<int>(1, count) = end; count = count + 1; len = 0; i = j; break; } } } else { len = 0; } } } } }
On top of that, we need to rotate the image segmentation function
void rotatepart(Mat& src) { vector<Mat> img1; //Store split pictures vector<Mat>::iterator imgiter; Mat srcclone = Mat::zeros(src.size(), src.type());//3 * column space Mat hist_ver; reduce(src / 255, hist_ver, 0, CV_REDUCE_SUM, CV_32S); //The image after preprocessing is first projected in abscissa, and the sum of all lines is taken. A value of 0 means that the line may be an interval int count = 0; int blankwidth = 18; int width = 5;//Greater than this interval is an area int totaln = hist_ver.cols; //Number of columns for image int reflen = 0; Mat locations = Mat::zeros(3, totaln, CV_32S);//3 * column space Find_begin_end_lencol(hist_ver, locations, count, blankwidth, reflen, 15); if (count > 2) count = 2; //cout << hist_ver << endl; double angle; for (int i = 0;i < count;i++) { //Parameters required for individual vertical projection//// int x = locations.at<int>(0, i); int y = 0; int bwid = locations.at<int>(2, i); int height = src.rows; Rect r2(x, y, bwid, height); Mat img = src(r2); //imshow(to_string(i+8 ), img); findangle(img, angle); // cout << angle << endl; Point2f center(img.rows / 2, img.cols / 2); Mat rot = getRotationMatrix2D(center, angle, 1); Mat aftero; warpAffine(img, aftero, rot, img.size()); // imshow(to_string(i+6), aftero); // imshow(to_string(i), img); Mat hist_ver1; reduce(aftero / 255, hist_ver1, 1, CV_REDUCE_SUM, CV_32S); //The image after preprocessing is first projected in abscissa, and the sum of all lines is taken. A value of 0 means that the line may be an interval transpose(hist_ver1, hist_ver1); //Column switching int count = 0; int width = 7;//Greater than this interval is an area int totaln = hist_ver1.cols; //hang number of images Mat locations1 = Mat::zeros(3, totaln, CV_32S);//3 * column space Find_begin_end_len(hist_ver1, locations1, count, width, 3); if (count >= 2) count = 2; // cout << hist_ver1; // cout << "count" << count << endl; for (int j = 0;j < count;j++) { int x = 0; int y = locations1.at<int>(0, j); int bwid = aftero.cols; int height = locations1.at<int>(2, j); Rect r3(x, y, bwid, height); img1.push_back(aftero(r3)); // imshow(to_string(i + j), aftero(r3)); } } //int i = 7; //for (imgiter = img1.begin(); imgiter != img1.end(); imgiter++) //{ // i++; // imshow(to_string(i), *imgiter); //} vector<Rect>rs; vector<Mat>srcroi; vector<Mat>mask; int xse, yse, xlen1, xlen2, ylen; yse = src.rows / 2; xse = src.cols * 2 / 3; xlen1 = xse; xlen2 = src.cols - xse; ylen = yse; int x1s, x2s, y1s, y2s; x1s = (xlen1 - img1[0].cols) / 2; x2s = xse + (xlen2 - img1[2].cols) / 2; y1s = (ylen - img1[0].rows) / 2; y2s = y1s + ylen; Rect rs1(x1s, y1s, img1[0].cols, img1[0].rows); rs.push_back(rs1); Rect rs2(x1s, y2s, img1[1].cols, img1[1].rows); rs.push_back(rs2); Rect rs3(x2s, y1s, img1[2].cols, img1[2].rows); rs.push_back(rs3); Rect rs4(x2s, y2s, img1[3].cols, img1[3].rows); rs.push_back(rs4); for (int i = 0;i < 4;i++) { srcroi.push_back(srcclone(rs[i])); mask.push_back(img1[i]); img1[i].copyTo(srcroi[i], mask[i]); } src = srcclone; //imshow("clone", src); }
Then the segmented image is projected at a certain angle, and then the position of the slant interval is obtained.
Upper function (the first part is to get the position roughly (which can be solved in most cases), and the second part is to move the point too small to the right or left):
oid getsep(Mat& line1, Mat& line2, double angle, int& count, int width, int minnum) { int N = line1.cols; vector<Rect>::iterator iter;//iterator int begin, len, end; Mat locationstemp = Mat::zeros(3, N, CV_32S);//3 * column space int n; // cout << line1 << endl; for (int i = 0;i < N;i++) { int n = line1.at<int>(0, i); if (n > 2) { begin = i; for (int j = i;j < N;j++) { n = line1.at<int>(0, j); if (j - begin > 35 && j == N - 1 && n >= 2 && j - begin <= 62) { end = j; len = end - begin; locationstemp.at<int>(0, count) = begin; locationstemp.at<int>(1, count) = end - len / 2; locationstemp.at<int>(2, count) = len / 2; count++; i = j; break; } if (j - begin > 62 && j == N - 1 && n >= 2) { end = j; len = end - begin; locationstemp.at<int>(0, count) = begin; locationstemp.at<int>(1, count) = end - 2 * len / 3; locationstemp.at<int>(2, count) = len / 3; count++; locationstemp.at<int>(0, count) = end - 2 * len / 3; locationstemp.at<int>(1, count) = end - len / 3; locationstemp.at<int>(2, count) = len / 3; count++; i = j; break; } if (n < minnum) { end = j; len = end - begin; if (len > width) { if (len > 32 && len < 62) { locationstemp.at<int>(0, count) = begin; locationstemp.at<int>(1, count) = end - len / 2; locationstemp.at<int>(2, count) = len / 2; count++; locationstemp.at<int>(0, count) = end - len / 2; locationstemp.at<int>(1, count) = end; locationstemp.at<int>(2, count) = len / 2; count++; } else if (len >= 62) { locationstemp.at<int>(0, count) = begin; locationstemp.at<int>(1, count) = end - 2 * len / 3; locationstemp.at<int>(2, count) = len / 3; count++; locationstemp.at<int>(0, count) = end - 2 * len / 3; locationstemp.at<int>(1, count) = end - len / 3; locationstemp.at<int>(2, count) = len / 3; count++; locationstemp.at<int>(0, count) = end - len / 3; locationstemp.at<int>(1, count) = end; locationstemp.at<int>(2, count) = len / 3; count++; } else { locationstemp.at<int>(0, count) = begin; locationstemp.at<int>(1, count) = end; locationstemp.at<int>(2, count) = len; count++; } //len = 0; } i = j; break; } } } } // cout << line2 << endl; int j = 1; line2.at<int>(0, 0) = locationstemp.at<int>(0, 0); line2.at<int>(1, 0) = locationstemp.at<int>(1, 0); line2.at<int>(2, 0) = locationstemp.at<int>(2, 0); for (int i = 1;i < count - 1;i++) { n = locationstemp.at<int>(2, i); line2.at<int>(0, j) = locationstemp.at<int>(0, i); line2.at<int>(1, j) = locationstemp.at<int>(1, i); line2.at<int>(2, j) = locationstemp.at<int>(2, i); if (n < 6 && n >= 1) { int beforelen, afterlen; beforelen = locationstemp.at<int>(0, i) - locationstemp.at<int>(1, i - 1); afterlen = locationstemp.at<int>(0, i + 1) - locationstemp.at<int>(1, i); if (beforelen < afterlen && (beforelen < 5 || afterlen < 5)) { j--; line2.at<int>(0, j) = locationstemp.at<int>(0, i - 1); line2.at<int>(1, j) = locationstemp.at<int>(1, i); line2.at<int>(2, j) = locationstemp.at<int>(1, i) - locationstemp.at<int>(0, i - 1); j++; } if (beforelen >= afterlen && (beforelen < 5 || afterlen < 5)) { line2.at<int>(0, j) = locationstemp.at<int>(0, i); line2.at<int>(1, j) = locationstemp.at<int>(1, i + 1); line2.at<int>(2, j) = locationstemp.at<int>(1, i + 1) - locationstemp.at<int>(0, i); i++; j++; } if (beforelen >= 5 && afterlen >= 5) { line2.at<int>(0, j) = locationstemp.at<int>(0, i); line2.at<int>(1, j) = locationstemp.at<int>(1, i); line2.at<int>(2, j) = locationstemp.at<int>(2, i); j++; } } else { j++; } } line2.at<int>(0, j) = locationstemp.at<int>(0, count - 1); line2.at<int>(1, j) = locationstemp.at<int>(1, count - 1); line2.at<int>(2, j) = locationstemp.at<int>(2, count - 1); //cout << locationstemp << endl<<line2<<endl; }
ok, let's put the main function in the end
int main() { double start = static_cast<double>(getTickCount()); Mat src, preimg, preimg1;//scr reading for source image int numcount = 0; int nums = 0; int n = 292; // ofstream outf("data1.txt"); for (n = 1;n < 929;n++) { //if (n == 292|| n==334||n==443 ||n==461|| n==487||n==548 ||n==550||n==571||n==685||n==740||n==744||n==751||n==836||n==731) // continue; vector<Mat> img; //Store split pictures vector<Mat>::iterator imgiter; vector<Point> pts; vector<Point>::iterator ptsiter; vector<Rect> r1; //Store split parameters vector<Rect>::iterator iter;//iterator src = imread("C:/Users/ggbond/Desktop/0617/ (" + to_string(n) + ")" + ".jpg"); //src = imread("C:/Users/ggbond/Desktop/20190628/ (" + to_string(n) + ").bmp"); //src = imread("C:/Users/ggbond/Desktop/cuowu/(" + to_string(n) + ").bmp"); if (src.empty()) { printf("could not load image...\r\n"); return -1; } preimg = predo(src); //Image preprocessing //preimg.convertTo(preimg1, CV_32S); //imshow("preimg", preimg); ///Split into 4 parts///// partimg(preimg, r1); if (r1.size() !=4) { r1.clear(); rotatepart(preimg); partimg(preimg, r1); } int i = 0; for (iter = r1.begin(); iter != r1.end(); iter++) { img.push_back(preimg(*iter)); //i++; ///*if(iter== r1.end())*/ //imshow(to_string(i), preimg(*iter)); } if (img.size() != 4) { printf("tehre is some problem in partimg...\r\n"); return -1; } /////Get a certain angle of projection///// int k = 0; for (imgiter = img.begin(); imgiter != img.end(); imgiter++) //for(int i=0;i<4;i++) { double angle = 86; int count = 0; int width = 2; int minnum = 1; //Mat line1 = Mat::zeros(1,img[i].cols, CV_32S); Mat line1 = Mat::zeros(1, (*imgiter).cols, CV_32S); Mat line2 = Mat::zeros(3, (*imgiter).cols, CV_32S); //If the pretreatment effect is not good enough, further treatment is needed here// //4. Some pictures change the parameters according to the situation to get better processing effect// if (k == 2) { seaprocess2((*imgiter)); width = 2; } if (k == 1) { seaprocess1((*imgiter)); } if (k == 0) { width = 2; seaprocess0((*imgiter)); minnum = 2; } slantreduce((*imgiter) / 255, line1, angle); getsep(line1, line2, angle, count, width, minnum); if (k == 1 && count > 2) count = 2; if (k == 0 && count > 7) count = 7; if (k == 2 && count > 7) count = 7; if (k == 3 && count > 3) count = 3; //cout << count << endl; vector<Point> pts1; for (int i = 0;i < count;i++) { int x = line2.at<int>(1, i); int tx = line2.at<int>(1, i) - (*imgiter).rows / tan(angle*PI / 180); int height = (*imgiter).rows; //pts1.push_back(Point(x,tx )); line(preimg, Point(x, 0) + r1[k].tl(), Point(tx, height) + r1[k].tl(), (0, 0, 255), 1, LINE_8); } //cout << line2 << endl; numcount = numcount + count; //if (k == 3) //{ //// imshow("zuizhongtu", preimg); // imwrite("C:/Users/ggbond/Desktop/rightimg0617/(" + to_string(n) + ").bmp", preimg); //} //imshow(to_string(k), (*imgiter)); k++; } if (numcount == 19) nums++; if (numcount != 19) { cout << "flse n" << n << endl; //outf << "flse" << n << endl; } cout << "The number of characters in this picture" << numcount << endl; cout << "Success times" << nums << endl; cout << "Execution times" << n << endl; //Outf < < number of characters in this picture < < numcount < < endl; //Outf < success times < nums < endl; //Outf < execution times < n < endl; numcount = 0; } //outf.close(); double end = static_cast<double>(getTickCount()); // getTickFrequency gets the number of clock cycles the CPU goes through in one second double run_time = (end - start) / getTickFrequency(); cout << "Running time"<<run_time << endl; waitKey(0); return 0; }
I'm so tired. The above is the first stage opencv learning achievement of our chicken. I hope you can give me some suggestions.
I'm impressed by the article
(principle of projection method)
