I. Problem Description
1) Pre-conditions
1. Configuring the anti-loss method of webrtc is FEC Only.
There are two anti-loss methods for webrtc: 1. NACK; 2. FEC. When starting configuration, you can choose to use NACK or FEC only or NACK and FEC at the same time.
Using both NACK and FEC, webrtc works by using only NACK when the network RTT delay is less than kLowRttNackMs (default 20ms). When the network delay is greater than _highRttNackMs, only FEC is used. RTT is between the two, using NACK and FEC mixed mode.
At present, in order to test the anti-loss performance of X264 FEC of webrtc, FEC Only mode is selected when starting configuration.
Refer to: Video Receiver:: SetVideo Protection
2. Using Network Emulator Client to configure the network packet loss model is random, and the probability of random packet loss is 5%.
2) Test cases
1. Establish X264 coded video connection to check the clarity and fluency of video.
2. Establish VP8 coded video connection to check the clarity and fluency of video.
3) Expected results
Video has no long-term Katon, at least X264 coding and decoding effect should be consistent with VP8.
4) Actual effect
X264 Carton is much more than VP8. According to log statistics,
Before VP8 decoding, 600 frames were observed continuously, the longest frame was dropped by 61 frames, and the shortest time for two consecutive cartons was 18 frames.
Before decoding, H264 has 600 frames in succession, and the longest frame is 86 frames. According to the sequence diagram, X264 will densely drop frames as long as packet loss is started, which is much worse than VP8.
II. Problem Analysis
1) Overview
Video data from network card to decode, after the following three aspects of judgment and processing:
1. RTP Packet Continuity Judgment. This section checks whether there is a packet loss based on the received message, and if there is, tries to parse the FEC message to see if it can be restored.
2. Video frame integrity judgment. This part can be referred to. "webrtc QOS Method 6 (Flower Screen Problem Solution)" To determine whether a frame_begin and frame_end exist, and whether the serial numbers of messages between frame_begin and frame_end are continuous, if there is the same unsatisfactory, the frame is not a complete frame, and the frame needs to be discarded.
3. Video inter-frame relationship judgment. Video frames have mutual reference relationship. If the I frame in front of the P frame is lost, even if there is a complete P frame, it can not be decoded. RtpFrameReference Finder:: ManageFrame Vp8, RtpFrameReference Finder:: ManageFrameGeneric function is to judge the reference relationship between frames.
2) Pre-frame statistics of video inter-frame relationship judgment
As can be seen from the above two graphs, before judging the inter-frame reference relationship, it is different.
Under the two models, the trend of packet loss is similar. At present, we can only check the frame integrity to judge the code, what is the difference in processing.
PacketBuffer:: There is a big difference between X264 and VP8 in FindFrames function.
1. The judgment of VP8 is relatively simple, as long as there are frame_begin and frame_end flags, and the messages between frame_begin and frame_end are continuous, it is considered as a complete frame.
2. The judgment of X264 adds IDR NAL and timestamp judgment on the basis of VP8.
std::vector<std::unique_ptr<RtpFrameObject>> PacketBuffer::FindFrames(
uint16_t seq_num) {
std::vector<std::unique_ptr<RtpFrameObject>> found_frames;
for (size_t i = 0; i < size_ && PotentialNewFrame(seq_num); ++i) {
size_t index = seq_num % size_;
sequence_buffer_[index].continuous = true;
// If all packets of the frame is continuous, find the first packet of the
// frame and create an RtpFrameObject.
if (sequence_buffer_[index].frame_end) {
size_t frame_size = 0;
int max_nack_count = -1;
uint16_t start_seq_num = seq_num;
// Find the start index by searching backward until the packet with
// the |frame_begin| flag is set.
int start_index = index;
size_t tested_packets = 0;
bool is_h264 = data_buffer_[start_index].codec == kVideoCodecH264;
bool is_h264_keyframe = false;
int64_t frame_timestamp = data_buffer_[start_index].timestamp;
while (true) {
++tested_packets;
frame_size += data_buffer_[start_index].sizeBytes;
max_nack_count =
std::max(max_nack_count, data_buffer_[start_index].timesNacked);
sequence_buffer_[start_index].frame_created = true;
if (!is_h264 && sequence_buffer_[start_index].frame_begin)
break;
if (is_h264 && !is_h264_keyframe) {
const RTPVideoHeaderH264& header =
data_buffer_[start_index].video_header.codecHeader.H264;
for (size_t i = 0; i < header.nalus_length; ++i) {
if (header.nalus[i].type == H264::NaluType::kIdr) {
is_h264_keyframe = true;
break;
}
}
}
if (tested_packets == size_)
break;
start_index = start_index > 0 ? start_index - 1 : size_ - 1;
// In the case of H264 we don't have a frame_begin bit (yes,
// |frame_begin| might be set to true but that is a lie). So instead
// we traverese backwards as long as we have a previous packet and
// the timestamp of that packet is the same as this one. This may cause
// the PacketBuffer to hand out incomplete frames.
// See: https://bugs.chromium.org/p/webrtc/issues/detail?id=7106
if (is_h264 &&
(!sequence_buffer_[start_index].used ||
data_buffer_[start_index].timestamp != frame_timestamp)) {
break;
}
--start_seq_num;
}
// If this is H264 but not a keyframe, make sure there are no gaps in the
// packet sequence numbers up until this point.
if (is_h264 && !is_h264_keyframe &&
missing_packets_.upper_bound(start_seq_num) !=
missing_packets_.begin()) {
uint16_t stop_index = (index + 1) % size_;
while (start_index != stop_index) {
sequence_buffer_[start_index].frame_created = false;
start_index = (start_index + 1) % size_;
}
return found_frames;
}
missing_packets_.erase(missing_packets_.begin(),
missing_packets_.upper_bound(seq_num));
found_frames.emplace_back(
new RtpFrameObject(this, start_seq_num, seq_num, frame_size,
max_nack_count, clock_->TimeInMilliseconds()));
}
++seq_num;
}
return found_frames;
}