Aiming at Bilibili's live video broadcasting function in response to the project's requirements, basically follow the example. It is found that Bilibili is implemented using a combination of Media Project and Virtual Display, which requires Android 5.0 Lollipop API 21 or more to be used.
Actually, it's officially provided. android-ScreenCapture This Sample already has the implementation and usage of MediaRecorder, as well as the Demo of recording screen to local file using MediaRecorder, from which we can all understand the use of these API s.
If we need live push stream, we need to customize MediaCodec, and then get the coded frame from MediaCodec, which saves us a lot of work in the process of collecting the original frame. But the problem arises because I did not know the structure of H264 file and the technology of FLV encapsulation carefully before, and I climbed many pits. After that, I will record them one by one, hoping to be helpful to the friends I used.
One Demo that has the greatest reference value for me in this project is the GitHub project of Yrom, a netizen. ScreenRecorder Demo implements video recording and saves the video stream as a local MP4 file (cough, is Yrom Bilibili's employee? ( -): smirk:. Here I will give a brief analysis of the implementation of the Demo, and then I will explain how I implemented it.
ScreenRecorder
Specific principles have been clearly stated in Demo's README:
- Display can be "projected" to a Virtual Display
- Create VirtualDisplay through MediaProjection Manager
- Virtual Display renders the image to Surface, which was created by MediaCodec
> mEncoder = MediaCodec.createEncoderByType(MIME_TYPE);
> ...
> mSurface = mEncoder.createInputSurface();
> ...
> mVirtualDisplay = mMediaProjection.createVirtualDisplay(name, mWidth, mHeight, mDpi, DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC, mSurface, null, null);
>
>
- MediaMuxer encapsulates image metadata obtained from MediaCodec and outputs it to MP4 files
> int index = mEncoder.dequeueOutputBuffer(mBufferInfo, TIMEOUT_US);
> ...
> ByteBuffer encodedData = mEncoder.getOutputBuffer(index);
> ...
> mMuxer.writeSampleData(mVideoTrackIndex, encodedData, mBufferInfo);
>
>
>
So in fact, on Android 4.4, you can create a Virtual Display through Display Manager, which can also achieve video recording, but ROOT is needed because of restrictions on permissions. (see DisplayManager.createVirtualDisplay() )
Demo is simple, two Java files:
- MainActivity.java
- ScreenRecorder.java
MainActivity
Class is only the entry to the implementation, the most important method is onActivityResult, because MediaProject needs to be opened from this method. But don't forget to initialize Media Project Manager first
@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
MediaProjection mediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, data);
if (mediaProjection == null) {
Log.e("@@", "media projection is null");
return;
}
// video size
final int width = 1280;
final int height = 720;
File file = new File(Environment.getExternalStorageDirectory(),
"record-" + width + "x" + height + "-" + System.currentTimeMillis() + ".mp4");
final int bitrate = 6000000;
mRecorder = new ScreenRecorder(width, height, bitrate, 1, mediaProjection, file.getAbsolutePath());
mRecorder.start();
mButton.setText("Stop Recorder");
Toast.makeText(this, "Screen recorder is running...", Toast.LENGTH_SHORT).show();
moveTaskToBack(true);
}
ScreenRecorder
This is a thread with clear structure. The initialization of MediaCodec, the creation of VirtualDisplay and the full implementation of cyclic coding are completed in the run() method.
Thread body
@Override
public void run() {
try {
try {
prepareEncoder();
mMuxer = new MediaMuxer(mDstPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
} catch (IOException e) {
throw new RuntimeException(e);
}
mVirtualDisplay = mMediaProjection.createVirtualDisplay(TAG + "-display",
mWidth, mHeight, mDpi, DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC,
mSurface, null, null);
Log.d(TAG, "created virtual display: " + mVirtualDisplay);
recordVirtualDisplay();
} finally {
release();
}
}
Initialization of MediaCodec
In this method, two key steps are taken: parameter configuration and start-up of encoder and creation of Surface.
private void prepareEncoder() throws IOException {
MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface); // The parameters that must be configured for recording screen
format.setInteger(MediaFormat.KEY_BIT_RATE, mBitRate);
format.setInteger(MediaFormat.KEY_FRAME_RATE, FRAME_RATE);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL);
Log.d(TAG, "created video format: " + format);
mEncoder = MediaCodec.createEncoderByType(MIME_TYPE);
mEncoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mSurface = mEncoder.createInputSurface(); // It needs to be created after createEncoderByType and before start(), and the source comments are clearly written.
Log.d(TAG, "created input surface: " + mSurface);
mEncoder.start();
}
Encoder realizes cyclic coding
The following code is the encoding process, because the author uses Muxer for video capture, so in resetOutputFormat method, the practical meaning is to transmit the encoded video parameter information to Muxer and start Muxer.
private void recordVirtualDisplay() {
while (!mQuit.get()) {
int index = mEncoder.dequeueOutputBuffer(mBufferInfo, TIMEOUT_US);
Log.i(TAG, "dequeue output buffer index=" + index);
if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
resetOutputFormat();
} else if (index == MediaCodec.INFO_TRY_AGAIN_LATER) {
Log.d(TAG, "retrieving buffers time out!");
try {
// wait 10ms
Thread.sleep(10);
} catch (InterruptedException e) {
}
} else if (index >= 0) {
if (!mMuxerStarted) {
throw new IllegalStateException("MediaMuxer dose not call addTrack(format) ");
}
encodeToVideoTrack(index);
mEncoder.releaseOutputBuffer(index, false);
}
}
}
private void resetOutputFormat() {
// should happen before receiving buffers, and should only happen once
if (mMuxerStarted) {
throw new IllegalStateException("output format already changed!");
}
MediaFormat newFormat = mEncoder.getOutputFormat();
// You can also get sps and pps here. See getSps PpsByteBuffer ()
Log.i(TAG, "output format changed.\n new format: " + newFormat.toString());
mVideoTrackIndex = mMuxer.addTrack(newFormat);
mMuxer.start();
mMuxerStarted = true;
Log.i(TAG, "started media muxer, videoIndex=" + mVideoTrackIndex);
}
Get the ByteBuffer of the sps pps, and note that the sps pps here are read-only
private void getSpsPpsByteBuffer(MediaFormat newFormat) {
ByteBuffer rawSps = newFormat.getByteBuffer("csd-0");
ByteBuffer rawPps = newFormat.getByteBuffer("csd-1");
}
Coding process of video frame on video recording
BufferInfo.flags represent currently encoded information, such as source annotations:
/**
* This indicates that the (encoded) buffer marked as such contains
* the data for a key frame.
*/
public static final int BUFFER_FLAG_KEY_FRAME = 1; // key frame
/**
* This indicated that the buffer marked as such contains codec
* initialization / codec specific data instead of media data.
*/
public static final int BUFFER_FLAG_CODEC_CONFIG = 2; // This state indicates that the current data is avcc, where sps pps can be obtained
/**
* This signals the end of stream, i.e. no buffers will be available
* after this, unless of course, {@link #flush} follows.
*/
public static final int BUFFER_FLAG_END_OF_STREAM = 4;
Implementation coding:
private void encodeToVideoTrack(int index) {
ByteBuffer encodedData = mEncoder.getOutputBuffer(index);
if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
// The codec config data was pulled out and fed to the muxer when we got
// the INFO_OUTPUT_FORMAT_CHANGED status.
// Ignore it.
// Configuration information (avcc) has been fed to Muxer in the previous resetOutputFormat(), which is not used here, but this step is a very important step in my project, because I need to manually implement SPS in advance, and the synthesis of PPS is sent to the streaming media server.
Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
mBufferInfo.size = 0;
}
if (mBufferInfo.size == 0) {
Log.d(TAG, "info.size == 0, drop it.");
encodedData = null;
} else {
Log.d(TAG, "got buffer, info: size=" + mBufferInfo.size
+ ", presentationTimeUs=" + mBufferInfo.presentationTimeUs
+ ", offset=" + mBufferInfo.offset);
}
if (encodedData != null) {
encodedData.position(mBufferInfo.offset);
encodedData.limit(mBufferInfo.offset + mBufferInfo.size); // Encoded Data is the encoded video frame, but note that the author does not distinguish between the key frame and the ordinary video frame, and writes the data to Muxer uniformly.
mMuxer.writeSampleData(mVideoTrackIndex, encodedData, mBufferInfo);
Log.i(TAG, "sent " + mBufferInfo.size + " bytes to muxer...");
}
}
The above is a general analysis of Screen Recorder Demo. Because of the hasty summary time, I have not conducted in-depth research on many details, so please read with suspicion. If there is a mistake or misunderstanding in the explanation, I hope you can help to point out. Thank you!