review
Ben delayed his sophomore electrical settings, and was lucky to join the team halfway. The author had not done camera and visual code before (AI electromagnetic), although there were various regrets in the end, but also temporarily learned a lot of new knowledge. Before the assessment, the foundation and development were completed successfully (1). Unfortunately, the problem was identified by changing the site after sealing the box.
The following problems occurred during the process:
- Low recognition frame rate during digital movement
- Recognition is too dependent on light intensity
- Trivial errors: not fully prepared before the race (camera AI, software teammates should cooperate to set up the code frame ahead of time), improper team division (no division of visual and control work, two cars not adjusted together [frame not advanced]), no inspection device after unpacking during the assessment (camera may change angle after sealing)
Ideas and Codes
Controlled by Infineon TC377, Camera Openmv Track, Openart Recognition
- Trace Code
The basic idea is as follows: the threshold of reddish shading, the binary midline of the track is fitted with linear regression, and the offset distance rho and the offset angle theta are used as err s; The average light of the brightness in the binarized image is used to determine whether it is a bifurcation element or not. (openmv has not been written before, the scheme to distinguish bifurcation is not very good, camera angle needs to be fixed and threshold value is better in each light. It is a new idea to set the area of interest on both left and right sides, and to distinguish bifurcation elements when there are more highlights on both sides of the center.)
import sensor, image, time, math
from image import SEARCH_EX, SEARCH_DS
from pyb import UART
uart = UART(3, 115200)
'Variable predefined'
global cnt_loss
global track
cnt_loss=0
track=0
THRESHOLD = (14, 65, -128, 127, 22, 127)
def err_revise():
rho_err = int(abs(line.rho())-img.width()/2)
if line.theta()>90:
theta_err = line.theta()-180
else:
theta_err = line.theta()
img.draw_line(line.line(), color = (0,0,255))
theta_err=-theta_err
err = theta_err + rho_err
return err
def track_judge():
global track
global cnt_loss
light = img.statistics().l_mean()
print(light)
if (light>=23):
track=2
elif light<23 and light>3:
track=1
else:
track=0
'main Part'
sensor.reset()
sensor.set_vflip(True)
sensor.set_hmirror(True)
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QQQVGA)
sensor.skip_frames(time = 2000)
clock = time.clock()
while(True):
clock.tick()
img = sensor.snapshot().binary([THRESHOLD])
line = img.get_regression([(100,100)], robust = True)
track_judge()
if (line):
err=err_revise()
if track==2:
uart.write( str(err)+'T' )
print('T cro')
elif track==1:
uart.write( str(err)+'F' )
print('F cro')
else:
uart.write( str(err)+'L' )
print('loss')
print(str(err)+'\n')
time.sleep_ms(30)
- Number recognition
Basic idea: Refer to the official open source routine flying by, look for a black border in the image, and input the image in the box into the model recognition. The first recognized number is sent to the MCU through the serial port, and then recognized number is sent to the MCU left and right according to the location of the black box center point.
Model training: basically follow the official tutorial of "Flying by Flying". At that time, Openart was already the second afternoon, and the time was tight. Good thing was the AI that had been electromagnetic before. The environment of the official training was configured with anconda, and the process also passed. But the recognition accuracy of the result training in the afternoon was very poor, and after the image enhancement with the training set of the big man (just add some other brightness pictures), the accuracy improved a lot, but at that time it felt especially affected by the ambient brightness. (Later, thinking carefully that the frame rate is actually too low, we should let the car stop at the intersection to recognize, the accuracy of static recognition was 90%+) in the final evaluation. The next night, the recognition is very unstable, and there is no combination of control. After rushing to set up a control framework to achieve 1 and 2 to enter the ward, more than two teammates suggested going back to sleep, and now I wonder if it was right or wrong. (The model in car race is transplanted after training of tensorflowlite. To tell the truth, the race also thought about its own training deployment at that time, but did not do the transplant model on openart. The transplant model is made by the government for fear of too much time. In fact, it can be done by using the TF Library of openmv. tf.load can load tensorflowlite model, just save it as tflite model after training)
'''Number recognition Openart Code'''
import pyb
import sensor, image, time, math
import os, nncu
from machine import UART
uart = UART(1, baudrate=115200)
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA) # we run out of memory if the resolution is much bigger...
sensor.set_brightness(300) # Set the brightness of the image to be bigger and brighter
sensor.skip_frames(time = 2000)
sensor.set_auto_gain(False) # must turn this off to prevent image washout...
sensor.set_auto_whitebal(False,(0,0x80,0)) # must turn this off to prevent image washout...
clock = time.clock()
net_path = "_1s_model_17_1.0000_xxxx.nncu" # Define Path to Model
labels = [line.rstrip() for line in open("/sd/labels_number.txt")] # Load Label
net = nncu.load(net_path, load_to_fb=True) # Load Model
first_num=0
is_send_num=0
while(True):
img = sensor.snapshot()
print("********** Start Detections **********")
for r in img.find_rects(threshold = 30000): # Search for Rectangles in Images
img.draw_rectangle(r.rect(), color = (255, 0, 0)) # Draw a rectangular outline to easily see the identified rectangular location on the IDE
img1 = img.copy(r.rect()) # Copy the image inside the rectangle
print(r.rect()[3])
if r.rect()[1]>10 and r.rect()[1]<140 and r.rect()[3]>65 and r.rect()[3]<90:
for obj in nncu.classify(net , img1, min_scale=1.0, scale_mul=0.5, x_overlap=0.0, y_overlap=0.0):
sorted_list = sorted(zip(labels, obj.output()), key = lambda x: x[1], reverse = True)
# Print the results with the highest accuracy
for i in range(1):
if first_num==0:
first_num=sorted_list[i][0]
###Send numbers to avoid Openmv Send out err Confusion, Openart Send letters corresponding to numbers#######
if str(first_num)=='1' and is_send_num==0:
uart.write('a')
print('1')
is_send_num=1
elif str(first_num)=='2' and is_send_num==0:
uart.write('b')
print('2')
is_send_num=1
elif str(first_num)=='3' and is_send_num==0:
uart.write('c')
print('3')
is_send_num=1
elif str(first_num)=='4' and is_send_num==0:
uart.write('d')
print('4')
is_send_num=1
elif str(first_num)=='5' and is_send_num==0:
uart.write('e')
print('5')
is_send_num=1
elif str(first_num)=='6' and is_send_num==0:
uart.write('f')
print('6')
is_send_num=1
elif str(first_num)=='7' and is_send_num==0:
uart.write('g')
print('7')
is_send_num=1
elif str(first_num)=='8' and is_send_num==0:
uart.write('h')
print('8')
sensor.set_brightness(400)
is_send_num=1
##################
print('sco'+str(first_num))
if sorted_list[i][0]==first_num:
if (r.rect()[0]<120):
uart.write('l')
print(str(first_num)+' on l')
else:
uart.write('r')
print(str(first_num)+' on r')
img.draw_string(r.rect()[0] + 20, r.rect()[1]-20, sorted_list[0][0],color = (0,0,255), scale = 2,mono_space=False)
- Control Code
Basic ideas:
1. Drug status marks:
Variable flag.drug-0.State of initial unpacked medication; 1.State after loading medication; 2.State of waiting for medication at ward stop (discarding line); 3.State of removing medication turning process; 4.State before reaching the last intersection after turning around; 5.State of returning to pharmacy after turning off the last intersection.
2. Corresponding control:
0.Parking, 1.Start and follow the PID route, get Openart information when you encounter an intersection to control left and right turns; 2. Parking; 3. Rotate 180 degrees in place; 4.PID routes, turn in the opposite direction when you come to an intersection; 5. Accelerate and follow the PID until you drop your car
3. Play Part One
Car 2 does not need to be identified in theory, it only needs to correspond to the control route according to the turn of Car 1.
Core Code:
1. Communication section (mv and MCU, art and MCU, dual car):
#include "headfile.h"
struct OPENMV mv,art;
char * p=mv.str;
int i=0;
/*
Function: openmv_get_str()
Variable description:
UART1-Communicate with openmv
UART0-Communicate with openart
UART2-Communicate with Car 2
@mv.data:Received characters are interrupted at each serial port
@mv.str:Store the complete data once in a string, T,F,L are the end signs sent once and correspond to T-intersection, F-not intersection, L-drop line respectively
@mv.err:Converting String Deviations to Floating Point Deviations
@mv.track:Corresponding T,L,F,2-is the intersection, 0-loses the line, 1-normal track
*/
void openmv_get_str()
{
if (uart_query(UART_1,&mv.data))
{
if (mv.data!='T' && mv.data!='F' && mv.data!='L' )
{
mv.str[i]=mv.data;
i++;
}
else if (mv.data=='T')
{
mv.str[i]='\0';
i=0;
mv.err = atof(mv.str);
memset(mv.str,0,sizeof(mv.str));
mv.track=2;
}
else if (mv.data=='F')
{
mv.str[i]='\0';
i=0;
mv.err = atof(mv.str);
memset(mv.str,0,sizeof(mv.str));
mv.track=1;
}
else if (mv.data=='L')
{
mv.str[i]='\0';
i=0;
mv.err = atof(mv.str);
memset(mv.str,0,sizeof(mv.str));
if (!art.con_left && !art.con_right) mv.track=0;
}
}
}
/*
Function: direc_control()
Variable description:
@art.con_left/right
State 1: (0 sec~cnt.turn_sec sec) Turn at a fixed duty cycle
State 2: (cnt.turn_sec sec~cnt.turn_sec*2 sec) During which the PID loops naturally and no line loss is judged (avoid arriving at the ward)
@cnt.left/right: Turn left the timer variable and interrupt 5 ms once.
@cnt.turn_time:The duration of the fixed differential of the turn and the period after the turn when the line is lost are not determined.
@cnt.turn_num:Number of turns (i.e. number of intersections encountered)
@flag.drug: Status of drugs
@flag.str: Whether to leave or not
@flag.is_judge: Play a variable of 1 that tells Car 2 that the mid-range ward intersection has been identified
@flag.send_direc:Play a variable of 1 that tells Car 2 the direction to turn
@flag.turn_memory[]: Write down the direction, and when you return to an intersection, start at the end of the array to control the direction
*/
void direc_control()
{
if (art.con_left)
{
cnt.left++;
if (cnt.left>cnt.turn_time*1000/5)
{
art.con_left=2;
if (cnt.left>cnt.turn_time*2000/5)
{
art.con_left=0;
cnt.left=0;
if (flag.drug<4&&flag.str)
{
if (!flag.is_judge)
{
flag.is_judge=1;
flag.send_direc='Z';
}
flag.turn_memory[cnt.turn_num]=1;
cnt.turn_num++;
}
else if(flag.drug>3)
{
cnt.turn_num--;
}
}
}
}
else if (art.con_right)
{
cnt.right++;
if (cnt.right>cnt.turn_second*1000/5)
{
art.con_right=2;
if (cnt.right>cnt.turn_second*2000/5)
{
/**Code in this if executes only once in a control (last 5ms interrupt entry)**/
art.con_right=0;
cnt.right=0;
if (flag.drug<4&&flag.str)
{
/*Play Part 1*/
if (!flag.is_judge)
{
flag.is_judge=1;
flag.send_direc='X';
}
/**********/
flag.turn_memory[cnt.turn_num]=2;
cnt.turn_num++;
}
else if(flag.drug>3)
{
cnt.turn_num--;
}
}
}
}
}
/*
Function: track_control()
Variable description:
@art.con_left/right: Described above
@art.num_judge:Recognized Numbers
@flag.turn_memory[]: Write down the direction, and when you return to an intersection, start at the end of the array to control the direction
Ideas: There are three main cases, 1. If 1 and 2 are identified, special treatment will be applied.
*/
void track_control()
{
if (mv.track==2 &&flag.drug!=5)
{
/*Identify 1 or 2*/
if (art.num_judge==1 || art.num_judge==2)
{
if (art.num_judge==1 && flag.drug!=4 && flag.drug!=5)
art.con_left=1;
else if (art.num_judge==1 && flag.drug==4)
{
art.con_right=1;
flag.drug=5;
}
else if (art.num_judge==2 && flag.drug!=4 && flag.drug!=5)
art.con_right=1;
else if (art.num_judge==2 && flag.drug==4)
{
art.con_left=1;
flag.drug=5;
}
}
/****************************/
/**Recognize Other Numbers*****/
else
{
/**If it's the first intersection (go straight to the next decision if there are no two)**/
else if(flag.drug==4 && cnt.turn_num == 2)
{
if(flag.turn_memory[1]==1)
art.con_right=1;
else if (flag.turn_memory[1]==2)
art.con_left=1;
}
/**If it's the last intersection**/
else if(flag.drug==4 && cnt.turn_num == 1 )
{
if(flag.turn_memory[0]==1)
art.con_right=1;
else if (flag.turn_memory[0]==2)
art.con_left=1;
flag.drug=5;
}
}
}
}
/*
Function: openart_get_str()
Variable description:
@art.data:Received single character (since the bottom of the three serial ports are read with a simpler blocking method, in addition to the openmv read deviation, the other two serial ports read only simple single characters and send such characters only for a certain serial port to avoid interfering with openmv communication whenever possible)
@art.num_judge:Recognized Numbers
*/
void openart_get_str()
{
if (uart_query(UART_0,&art.data))
{
if (art.data=='l')
{
art.con_left=1;
}
else if (art.data=='r')
{
art.con_right=1;
}
else if (art.data=='a')
{
art.num_judge=1;
}
else if (art.data=='b')
{
art.num_judge=2;
}
else if (art.data=='c')
{
art.num_judge=3;
}
else if (art.data=='d')
{
art.num_judge=4;
}
else if (art.data=='e')
{
art.num_judge=5;
}
else if (art.data=='f')
{
art.num_judge=6;
}
else if (art.data=='g')
{
art.num_judge=7;
}
else if (art.data=='h')
{
art.num_judge=8;
}
}
}
2. Motor Control
/*
Function: motor_control()
Variable description:
@enc.speed_contorl_out:Speed Ring Output
@enc.dif:Directional Ring Output Differential Speed
@enc.left_pwm/right:Left and right speed
@cnt.round_second:In-place rotation duration
*/
void motor_control()
{
if (flag.drug==5)
{
enc.left_pwm=enc.speed_control_out+2000-enc.dif;//steer.angle is the motor differential
enc.right_pwm=enc.speed_control_out+2000+enc.dif;
}
else
{
enc.left_pwm=enc.speed_control_out+0-enc.dif;//steer.angle is the motor differential
enc.right_pwm=enc.speed_control_out+0+enc.dif;
}
if (!flag.man_swi)
{
if (flag.drug==3)
{
cnt.turn_round++;
if (cnt.turn_round>cnt.round_second*1000/5)
{
cnt.turn_round=0;
flag.drug=4;
}
motor(3000);
motor2(3000);
}
else if(art.con_left==1)
{
motor(1000);
motor3(4000);
}
else if (art.con_right==1)
{
motor(4000);
motor3(1000);
}
else if (mv.track==0 && (flag.drug==5 || flag.drug==2) )
{
motor(0);
motor3(0);
motor2(0);
motor1(0);
}
else
{
if (enc.left_pwm>0)
{
motor(enc.left_pwm);
motor1(0);
}
else
{
motor(0);
motor1(-enc.left_pwm);
}
if (enc.right_pwm>0)
{
motor2(0);
motor3(enc.right_pwm*1.2);
}
else
{
motor2(-enc.right_pwm*1.2);
motor3(0);
}
}
}
}
3. Interruption
IFX_INTERRUPT(cc61_pit_ch0_isr, 0, CCU6_1_CH0_ISR_PRIORITY)
{
enableInterrupts();//Turn on interrupt nesting
PIT_CLEAR_FLAG(CCU6_1, PIT_CH0);
//Automatic Tracing--AI/PID
if (!flag.man_swi)
{
/***************PID*************/
/*Send signal to car 2*/
Send_to_car2();
/*Peripheral Processing*/
reeds_get();//Check if the medicine is loaded
if_put_drug();//Do you put medication on
/*Bright red light - ward lose line*/
if (flag.drug==1 && mv.track==0)
{
flag.drug=2;
flag.led=1;
}
if_get_drug();//Is drug removed
/*Green light - Drug store lose line*/
if (flag.drug==5 && mv.track==0)
flag.led=2;
led_control();//Tip light control
/*Camera Processing*/
openmv_get_str();
if (!art.left_control && !art.right_control)
{
if (flag.drug<3) openart_get_str();
track_control()
}
direc_control();
PID_steer_trace();
/**Prepare, depart **/
if (flag.str)
{
if (flag.mot_con) motor_control();
// if (flag.ste_con) steer_control(); A four-wheeled car doesn't turn well in place and a ride wastes a lot of time
}
/**Count**/
Count();
}
/************Manual Test*********/
else
{
openmv_get_str();
if (flag.str)
{
if (flag.mot_con) motor_control();
// if (flag.ste_con) steer_control();
}
/**Count**/
Count();
}
}
4.Car 2 Code
/**Get Car 1 Directive**/
void car2_get_str()
{
if (uart_query(UART_2,&getcar1.data))
{
if (getcar1.data=='Z' )
{
flag.turn_direc=1;
flag.str=1;
}
else if (getcar1.data=='X')
{
flag.turn_direc=2;
flag.str=1;
}
else if (getcar1.data=='S')
{
flag.get_turn=1;
}
}
}
/**Direction Control**/
void track_control()
{
if (mv.track==2 &&flag.drug!=5)
{
if (flag.turn_direc==1)
{
art.con_left=1;
}
else if (flag.turn_direc==2)
{
art.con_right=1;
}
}
}
void direc_control()
{
//Same Car 1
}
void motor_control()
{
//Same Car 1
}
/***Interruption section***/
IFX_INTERRUPT(cc61_pit_ch0_isr, 0, CCU6_1_CH0_ISR_PRIORITY)
{
enableInterrupts();//Turn on interrupt nesting
PIT_CLEAR_FLAG(CCU6_1, PIT_CH0);
//Automatic Tracing--AI/PID
if (!flag.man_swi)
{
/***************PID*************/
/*Receive Main Car Communication*/
car2_get_str();
/*Peripheral Processing*/
led_control();
flag.drug=0;
/*Camera Processing*/
openmv_get_str();
if (cnt.left==0 && cnt.right==0)
{
track_control();
}
direc_control();
PID_steer_trace();
/**Prepare, depart **/
if (flag.str)
{
if (flag.mot_con) motor_control();
}
/**Count**/
Count();
}
}
Optimizable Points
- Recognition frame rate is too low during digital movement, resulting in inaccurate recognition:
An effective solution is to add the judgment that the cross parking 2s enters the recognition state to the control, exit the recognition state after the recognition and control the turn.
Unfortunately: at that time, the time was too busy, the physical condition was not good, people stayed up late and felt that the overall time would exceed. In fact, as long as you control the motor to speed up more, and experts did not emphasize the time in the assessment. This is really my regret. At that time, people were numb and didn't think well. It was also too technical and difficult to say a word. - Recognition is too dependent on light intensity
In fact, I still have some doubts about this, because the above problem is the real main factor, and the recognition accuracy of numbers in static state is more than 90%: (there is hardly any missed recognition of handheld numbers that day). Of course, the impact after the change of site is also very big (brightness at 8 am is not the same as in the afternoon). It is really necessary to write a filter algorithm to make the camera adapt to the uniform light in various scenes. The direct light from the sun asks the students who adjust the camera to solve the problem, so they can only pull the curtains.
Unfortunately: Too much time has been delayed at this point. Control and communication have been written for about 4 hours. Recognition has been ineffective, but I did not expect that the main direction of thinking is wrong. Instead of continuously improving the training set to improve light and dark recognition and reducing the model complexity to improve the frame rate, it is better to make big changes in control. Stopping at the cross is the core of recognition. Later, it was seen that students matching openmv with templates had a low recognition rate.
End
- At that time, there was not much time to take pictures for remembering, then the box was unpacked and optimized, making point effect pictures, video playback blog reminiscence. This may be the last time the university has participated in the competition, which is a pity in the end.
Personal is only responsible for visual and control software part, hardware has the opportunity to ask your teammates to release. After that, I also plan to organize the code for each project to put on the webdisks, blogs, duplicate disks, organize more, keep growing, the future may be expected!
Identify big guys (hate late meeting):
https://zhuanlan.zhihu.com/p/391216590
openmv Manual in Chinese
https://docs.singtown.com/micropython/zh/latest/openmvcam/index.html
openart Open Source Tutorial:
http://mp.weixin.qq.com/s?__biz=MzAxMjQxNjEyMw==&mid=2247487040&idx=1&sn=4d47976b398b3c92811a29356ada34b9&chksm=9bb36b54acc4e242edf935986c97ee1f6363dffc86678faf64c65903476f8c27504e1a392120&mpshare=1&scene=23&srcid=1203gAB1ALoBvqjYxJd7qp5c&sharer_sharetime=1638461358289&sharer_shareid=b8ab17ea4161b8f0d5c69ddb66ea7563#rd
rule
