When Beauty is turned on, double-stream is turned on. Small-stream blinks when viewed remotely

GPUImage is used in the project for cosmetic treatment, which adds whitening, polishing and bright light, but when the cosmetic effect is turned on, if you only use the big stream to view the video, the video looks normal and the picture is normal, but if you use the small stream to view the cosmetic image, the black blocks in the video will flash occasionally, similar to the large mosaic effect, and then analyze if you willBeauty effect is off, small stream video picture is normal, video processing flow: the collected video picture is processed by YUV or RGB Beauty Rendering, then the frame of the picture is passed to WebRTC for encoding, and then up to the server, down to remote viewing.

Guess, since you turn off the beauty picture, the small stream picture is not a problem, so it may be a problem with the beauty treatment. So, annotate the beauty related code and find the picture black screen, but there are still black blocks flashing, so the probability is not the problem caused by the beauty treatment.

Continuing to analyze the code, we found that the video frame data of the same address were processed in the process of beauty first and then encoding, but beauty and encoding were processed in two different asynchronous threads. This will lead to inconsistency between the data of encoding and beauty processing, because beauty can change YUV data, so it may cause black block blinking problem when streaming.A method of copy ing video data was added to keep the data processed by encoding consistent with the data processed by beauty, and then the detection found that the stream no longer flickered, which solved the problem (incredibly happy! Incredibly excited!).

copy Method of Video Data Address
/// RGB/BGR buffer copy
+ (CVPixelBufferRef)RGBBuffereCopyWithPixelBuffer:(CVPixelBufferRef)pixelBuffer {
    // Get pixel buffer info
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
    int bufferWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    int bufferHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
    uint8_t *baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer);
    OSType pixelFormat = kCVPixelFormatType_32BGRA;
    
    // Copy the pixel buffer
    CVPixelBufferRef pixelBufferCopy = NULL;
    CFDictionaryRef empty = CFDictionaryCreate(kCFAllocatorDefault, NULL, NULL, 0, &kCFTypeDictionaryKeyCallBacks, &kCFTypeDictionaryValueCallBacks); // our empty IOSurface properties dictionary
    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                             empty, kCVPixelBufferIOSurfacePropertiesKey,
                             nil];
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, bufferWidth, bufferHeight, pixelFormat, (__bridge CFDictionaryRef) options, &pixelBufferCopy);
    if (status == kCVReturnSuccess) {
        CVPixelBufferLockBaseAddress(pixelBufferCopy, 0);
        uint8_t *copyBaseAddress = CVPixelBufferGetBaseAddress(pixelBufferCopy);
        memcpy(copyBaseAddress, baseAddress, bufferHeight * bytesPerRow);
        
        CVPixelBufferUnlockBaseAddress(pixelBufferCopy, 0);
    }else {
        SmoothLogE("RGBBuffereCopyWithPixelBuffer :: failed");
    }
    
    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
    return pixelBufferCopy;
}

However, it's too early to be happy. During the test, we found that the stability deteriorated because we copied a piece of data, ran a little longer, memory grew, and had a bad experience on the old iPhone, so we went on to research...

Based on the phenomenon of flickering black patches and the canvas phenomenon after commenting on the beauty code, it feels like the coded picture is not the same size as the beauty picture, so the code is analyzed and a method of coding an adapter for the same broadcast is found (int SimulcastEncoderAdapter::Encode)For encoding, one of the methods is to zoom in and out of the video picture, so try changing this method.

int SimulcastEncoderAdapter::Encode(
    const VideoFrame& input_image,
    const std::vector<VideoFrameType>* frame_types) {
  RTC_DCHECK_RUN_ON(&encoder_queue_);

  if (!Initialized()) {
    return WEBRTC_VIDEO_CODEC_UNINITIALIZED;
  }
  if (encoded_complete_callback_ == nullptr) {
    return WEBRTC_VIDEO_CODEC_UNINITIALIZED;
  }

  // All active streams should generate a key frame if
  // a key frame is requested by any stream.
  bool send_key_frame = false;
  if (frame_types) {
    for (size_t i = 0; i < frame_types->size(); ++i) {
      if (frame_types->at(i) == VideoFrameType::kVideoFrameKey) {
        send_key_frame = true;
        break;
      }
    }
  }
  for (size_t stream_idx = 0; stream_idx < streaminfos_.size(); ++stream_idx) {
    if (streaminfos_[stream_idx].key_frame_request &&
        streaminfos_[stream_idx].send_stream) {
      send_key_frame = true;
      break;
    }
  }

  int src_width = input_image.width();
  int src_height = input_image.height();
  for (size_t stream_idx = 0; stream_idx < streaminfos_.size(); ++stream_idx) {
    // Don't encode frames in resolutions that we don't intend to send.
    if (!streaminfos_[stream_idx].send_stream) {
      continue;
    }

    std::vector<VideoFrameType> stream_frame_types;
    if (send_key_frame) {
      stream_frame_types.push_back(VideoFrameType::kVideoFrameKey);
      streaminfos_[stream_idx].key_frame_request = false;
    } else {
      stream_frame_types.push_back(VideoFrameType::kVideoFrameDelta);
    }

    int dst_width = streaminfos_[stream_idx].width;
    int dst_height = streaminfos_[stream_idx].height;
    // If scaling isn't required, because the input resolution
    // matches the destination or the input image is empty (e.g.
    // a keyframe request for encoders with internal camera
    // sources) or the source image has a native handle, pass the image on
    // directly. Otherwise, we'll scale it to match what the encoder expects
    // (below).
    // For texture frames, the underlying encoder is expected to be able to
    // correctly sample/scale the source texture.
    // TODO(perkj): ensure that works going forward, and figure out how this
    // affects webrtc:5683.
      /**
       If scaling is not required, the image will be transferred directly because the input resolution matches the target, or the input image is empty (for example, a keyframe request from an encoder with an internal camera source), or the source image has a native handle. Otherwise, we will scale it to match what the encoder expects (as shown below).For texture frames, the underlying encoder should be able to sample/scale the source texture correctly. TODO (perkj): Ensure that this work continues and find out how this affects webrtc:5683.
       Encode without scaling, keeping the size stream identical, commenting out the RTCVideoEncoderH264.mm encoding method - (NSInteger) encode:(RTCVideoFrame *)
       codecSpecificInfo:(nullable id<RTCCodecSpecificInfo>)codecSpecificInfo
              frameTypes:(NSArray<NSNumber *> *)frameTypes Medium width and high judgment:
//  RTC_DCHECK_EQ(frame.width, _width);
//  RTC_DCHECK_EQ(frame.height, _height);
Can solve the problem of small streams flickering
       */
//    if ((dst_width == src_width && dst_height == src_height) /*||
//        input_image.video_frame_buffer()->type() ==
//            VideoFrameBuffer::Type::kNative*/) {
      int ret = streaminfos_[stream_idx].encoder->Encode(input_image,
                                                         &stream_frame_types);
      if (ret != WEBRTC_VIDEO_CODEC_OK) {
        return ret;
      }
//    } else {
//      rtc::scoped_refptr<I420Buffer> dst_buffer =
//          I420Buffer::Create(dst_width, dst_height);
//      rtc::scoped_refptr<I420BufferInterface> src_buffer =
//          input_image.video_frame_buffer()->ToI420();
//      libyuv::I420Scale(src_buffer->DataY(), src_buffer->StrideY(),
//                        src_buffer->DataU(), src_buffer->StrideU(),
//                        src_buffer->DataV(), src_buffer->StrideV(), src_width,
//                        src_height, dst_buffer->MutableDataY(),
//                        dst_buffer->StrideY(), dst_buffer->MutableDataU(),
//                        dst_buffer->StrideU(), dst_buffer->MutableDataV(),
//                        dst_buffer->StrideV(), dst_width, dst_height,
//                        libyuv::kFilterBilinear);
//
//      // UpdateRect is not propagated to lower simulcast layers currently.
//      // TODO(ilnik): Consider scaling UpdateRect together with the buffer.
//      VideoFrame frame(input_image);
//      frame.set_video_frame_buffer(dst_buffer);
//      frame.set_rotation(webrtc::kVideoRotation_0);
//      frame.set_update_rect(
//          VideoFrame::UpdateRect{0, 0, frame.width(), frame.height()});
//      int ret =
//          streaminfos_[stream_idx].encoder->Encode(frame, &stream_frame_types);
//      if (ret != WEBRTC_VIDEO_CODEC_OK) {
//        return ret;
//      }
//    }
  }

  return WEBRTC_VIDEO_CODEC_OK;
}

Then the effect test found that the problem of small stream flicker was solved, and the memory usage was basically the same as before, so it can be said that the problem was completely solved. (Unhappy, finally solved a difficult problem, persevere is victory!!)