In the past two years, I have been engaged in the development of the video industry. Overtime is also serious. I haven't written articles for a long time. Recently, I have a little time to write some articles to record some problems encountered in the development and solutions.

Video conferencing projects, of course, are inseparable from audio and video, and often deal with WebRTC, including compiling WebRTC, modifying the source code, as well as C++ and OC mixed development, JS interaction, but also tried RN development to avoid some problems, and so on. There are many things, the road is not easy to walk, overtime is also common, but now the project base. This is stable, has been online, user feedback is also very good! Look back at the road, although bumpy, but also full of harvest!

To get to the point, there are shared functions in the project, ReplayKit for screen sharing, but Broadcast Extension is used for out-of-application screen sharing. This extension is another independent process, involving the interaction between host App and extended App's signaling. Fortunately, it saves a lot of things when used by a better third party. Completed screen sharing! If you have friends involved in this knowledge, you can communicate with Ha!

In addition to screen sharing, we have to share web pages, share web pages, and start to be confused! Later, the solution is that the sharing end uses WK WebView and the viewing end watches video streams!OK, then the problem is how to collect rendering data of a UIView into a Buffer, the first big question is how to use the video recorder in the application. This is a solution, but if you record the screen, you can collect all the content on the screen. It's not very friendly. Users just want to share this page. Others don't want the viewers to see it. It's really troublesome. We checked a lot of information, and it's not particularly suitable.

The final solution is to create CVPixel BufferRef pxbuffer and get the basic address of the pixel.

void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
Then the context is created with the memory pointed to by pxdata as the container of the context data storage, and the context is rendered [self.webView.layer renderInContext:context].
In this way, the rendering data on the webView view is converted into a frame of CVPixel BufferRef data, and then the timer is set according to the frame rate to continuously obtain the rendering data of the webView and convert it into the required pixel buffer data!

The specific code is as follows

- (CVPixelBufferRef)CVPixelBufferRef {
    
    CGSize size = self.frame.size;
    NSDictionary *options = @{(NSString*)kCVPixelBufferCGImageCompatibilityKey : @YES,
                              (NSString*)kCVPixelBufferCGBitmapContextCompatibilityKey : @YES,
                              (NSString*)kCVPixelBufferIOSurfacePropertiesKey: [NSDictionary dictionary]};
    CVPixelBufferRef pxbuffer = NULL;
    
    CGFloat frameWidth = size.width;
    CGFloat frameHeight = size.height;
    
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault,
                                          frameWidth,
                                          frameHeight,
                                          kCVPixelFormatType_32ARGB,
                                          (__bridge CFDictionaryRef) options,
                                          &pxbuffer);
    
    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
    
    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);
    
    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    
    CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, CVPixelBufferGetBytesPerRow(pxbuffer), rgbColorSpace, kCGImageAlphaPremultipliedFirst);
    
    NSParameterAssert(context);
    
    CGContextConcatCTM(context, CGAffineTransformMakeRotation(0));
    CGAffineTransform flipVertical = CGAffineTransformMake( 1, 0, 0, -1, 0, frameHeight);
    CGContextConcatCTM(context, flipVertical);
    
//    CGAffineTransform flipHorizontal = CGAffineTransformMake( -1.0, 0.0, 0.0, 1.0, frameWidth, 0.0 );
//    CGContextConcatCTM(context, flipHorizontal);


    [self.webView.layer renderInContext:context];
    
    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);
    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
    
    return pxbuffer;
}

This may not be the best way, but it does solve the pressing problem, share it, provide a solution for the same needs of small partners, or small partners have a better way to share it, you can learn to exchange.

Or my brief book