Article catalog
Android live streaming process: the mobile phone collects video / audio data, video data is encoded by H.264, audio data is encoded by AAC, and finally the audio and video data are packed into RTMP packets, which are uploaded to RTMP server by RTMP protocol;
In Android terminal, video data collection is mainly completed on mobile terminal, and video data is transferred to JNI. In NDK, x264 is used to convert image to H.264 video. Finally, H.264 video is packaged into RTMP data package and uploaded to RTMP server;
This blog introduces the following content: Java layer transfers NV21 format data collected by Camera to JNI layer, and uses x264 encoder to encode NV21 image data into H.264 video data in JNI;
This blog mainly encapsulates AVC sequence header data, including frame type, AVC data type, synthesis time, version information, coding specification, NALU length, SPS number, SPS length, SPS data, PPS number, PPS length, PPS data into RTMP package;
1, Basic package data format description
1. This is the complete video tag data content: This is the complete video tag data in FLV;
0x00000182 : 09 00 00 2E 00 00 00 00 0x0000018a : 00 00 00 17 00 00 00 00 0x00000192 : 01 64 00 32 FF E1 00 19 0x0000019a : 67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68 01 00 05 68 E9 7B 2C 0x000001ba : 8B 00 00 00 39
2. Tag header: the first 111111 bytes are tag header data, which stores 111111 bytes of information such as tag type, tag data size, time stamp, time stamp extension bit, stream number, etc;
0x00000182 : 09 00 00 2E 00 00 00 00 0x0000018a : 00 00 00
3. label data as like as two peas: key points: This is what the blog wants to encapsulate, basically encapsulating a RTMP package that is exactly the same format.
17 00 00 00 00 0x00000192 : 01 64 00 32 FF E1 00 19 0x0000019a : 67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68 01 00 05 68 E9 7B 2C 0x000001ba : 8B 00 00 00 39
Reference blog: refer to the two previous blogs that analyze RTMP data format, and analyze FLV video data format which is almost consistent with RTMP format;
These two blogs must understand the FLV video tag data format, so as to understand the RTMP package content written today;
2, General description of packaged SPS PPS data
1. Data example:
17 00 00 00 00 0x00000192 : 01 64 00 32 FF E1 00 19 0x0000019a : 67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68 01 00 05 68 E9 7B 2C 0x000001ba : 8B 00 00 00 39
- 17 frame type, 1 byte
- 00 data type, 1 byte
- 00 00 00 synthesis time, 3 bytes
- 01 version information, 1 byte
- 64 00 32 encoding rule, 3 bytes
- FF NALU length, 1 byte
- Number of E1 SPS, 1 byte
- 00 19 SPS length, 2 bytes
13 bytes of data as of current location
- spsLen byte data, here is 25 bytes
67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68
- 01 PPS, 1 byte
- 00 05 PPS length, 2 bytes
- PPS data of ppsLen bytes
68 E9 7B 2C
0x000001ba : 8B
- The following 00 00 39 is the total length of the video tag, which can not be encapsulated in the RTMP tag;
2. Calculate the size of the whole SPS and PPS data:
① Package header: frame type, data type, synthesis time, version information, coding rule, NALU length, 101010 bytes in total;
② Package SPS data: number of SPS, length of SPS, SPS data, 1+2+spsLen1 + 2 + spsLen1+2+spsLen bytes respectively;
③ Package PPS data: PPS number, PPS length, PPS data, with 1+2+ppsLen1 + 2 + ppsLen1+2+ppsLen bytes respectively;
int rtmpPackagesize = 10 + 3 + spsLen + 3 + ppsLen;
3, Package header data
In RTMP packet, encapsulate frame type, data type, synthesis time, version information, coding rule, NALU length, with a total of 101010 bytes;
// Frame type data: divided into two parts; // The first 4 bits represent frame type, 1 represents key frame, and 2 represents normal frame // The last 4 bits represent encoding type, and 7 represent AVC video encoding rtmpPacket->m_body[nextPosition++] = 0x17; // Data type, 00 for AVC sequence header rtmpPacket->m_body[nextPosition++] = 0x00; // Synthesis time, general setting 00 00 rtmpPacket->m_body[nextPosition++] = 0x00; rtmpPacket->m_body[nextPosition++] = 0x00; rtmpPacket->m_body[nextPosition++] = 0x00; // Version information rtmpPacket->m_body[nextPosition++] = 0x01; // Coding specification rtmpPacket->m_body[nextPosition++] = sps[1]; rtmpPacket->m_body[nextPosition++] = sps[2]; rtmpPacket->m_body[nextPosition++] = sps[3]; // NALU length rtmpPacket->m_body[nextPosition++] = 0xFF;
4, Encapsulating SPS data
Package SPS data into RTMP data package, including the number of SPS, SPS length, SPS data;
// Number of SPS rtmpPacket->m_body[nextPosition++] = 0xE1; // SPS length, 2 bytes // Set high order of length rtmpPacket->m_body[nextPosition++] = (spsLen >> 8) & 0xFF; // Set the low order of length rtmpPacket->m_body[nextPosition++] = spsLen & 0xFF; // Copy SPS data // Copy SPS data to rtmppacket - > m_ In the body [nextposition] address memcpy(&rtmpPacket->m_body[nextPosition], sps, spsLen); // Accumulated SPS length information nextPosition += spsLen;
5, Package PPS data
Package PPS data into RTMP data package, including PPS number, PPS length, PPS data;
// Number of PPS rtmpPacket->m_body[nextPosition++] = 0x01; // Length of PPS data, 2 bytes // Set high order of length rtmpPacket->m_body[nextPosition++] = (ppsLen >> 8) & 0xFF; // Set the low order of length rtmpPacket->m_body[nextPosition++] = (ppsLen) & 0xFF; // Copy SPS data memcpy(&rtmpPacket->m_body[nextPosition], pps, ppsLen);
6, Set other parameters of RTMP packet
Set RTMP packet type, RTMP packet length, RTMP channel, time stamp and other information;
// Set RTMP package type and video type data rtmpPacket->m_packetType = RTMP_PACKET_TYPE_VIDEO; // Set RTMP package length rtmpPacket->m_nBodySize = rtmpPackagesize; // Assign RTMP channels, assign at will rtmpPacket->m_nChannel = 10; // Set the video time stamp. If it is SPP PPS data, there is no time stamp rtmpPacket->m_nTimeStamp = 0; // Set the absolute time, and assign 0 to SPS PPS rtmpPacket->m_hasAbsTimestamp = 0; // Set the header type, and set one at will rtmpPacket->m_headerType = RTMP_PACKET_SIZE_MEDIUM;
7, SPS PPS data encapsulation code example
/** * Send SPS / PPS data to RTMP server * @param sps SPS data * @param pps PPS data * @param spsLen SPS length * @param ppsLen PPS length */ void VedioChannel::sendSpsPpsToRtmpServer(uint8_t *sps, uint8_t *pps, int spsLen, int ppsLen) { // Create an RTMP packet and store all data in the RTMP packet RTMPPacket *rtmpPacket = new RTMPPacket; /* Calculate the size of the whole SPS and PPS data Data example: 17 00 00 00 00 0x00000192 : 01 64 00 32 FF E1 00 19 0x0000019a : 67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68 01 00 05 68 E9 7B 2C 0x000001ba : 8B 00 00 00 39 17 Frame type, 1 byte 00 Data type, 1 byte 00 00 00 Synthesis time, 3 bytes 01 Version information, 1 byte 64 00 32 Encoding rules, 3 bytes FF NALU Length, 1 byte E1 SPS Number, 1 byte 00 19 SPS Length, 2 bytes 13 bytes of data as of current location spsLen Byte data, 25 bytes here 67 64 00 32 AC D9 80 78 0x000001a2 : 02 27 E5 84 00 00 03 00 0x000001aa : 04 00 00 1F 40 3C 60 C6 0x000001b2 : 68 01 PPS Number, 1 byte 00 05 PPS Length, 2 bytes ppsLen Bytes of PPS data 68 E9 7B 2C 0x000001ba : 8B The next 00 00 00 39 is the total length of the video tag There is no need to package in RTMP label */ int rtmpPackagesize = 10 + 3 + spsLen + 3 + ppsLen; // Allocate memory for RTMP packets RTMPPacket_Alloc(rtmpPacket, rtmpPackagesize); // Record the next index location to write data to int nextPosition = 0; // Frame type data: divided into two parts; // The first 4 bits represent frame type, 1 represents key frame, and 2 represents normal frame // The last 4 bits represent the encoding type and 7 represents the AVC video encoding rtmpPacket->m_body[nextPosition++] = 0x17; // Data type, 00 for AVC sequence header rtmpPacket->m_body[nextPosition++] = 0x00; // Synthesis time, general setting 00 00 rtmpPacket->m_body[nextPosition++] = 0x00; rtmpPacket->m_body[nextPosition++] = 0x00; rtmpPacket->m_body[nextPosition++] = 0x00; // Version information rtmpPacket->m_body[nextPosition++] = 0x01; // Coding specification rtmpPacket->m_body[nextPosition++] = sps[1]; rtmpPacket->m_body[nextPosition++] = sps[2]; rtmpPacket->m_body[nextPosition++] = sps[3]; // NALU length rtmpPacket->m_body[nextPosition++] = 0xFF; // Number of SPS rtmpPacket->m_body[nextPosition++] = 0xE1; // SPS length, 2 bytes // Set high order of length rtmpPacket->m_body[nextPosition++] = (spsLen >> 8) & 0xFF; // Set the low order of length rtmpPacket->m_body[nextPosition++] = spsLen & 0xFF; // Copy SPS data // Copy SPS data to rtmppacket - > m_ In the body [nextposition] address memcpy(&rtmpPacket->m_body[nextPosition], sps, spsLen); // Accumulated SPS length information nextPosition += spsLen; // Number of PPS rtmpPacket->m_body[nextPosition++] = 0x01; // Length of PPS data, 2 bytes // Set high order of length rtmpPacket->m_body[nextPosition++] = (ppsLen >> 8) & 0xFF; // Set the low order of length rtmpPacket->m_body[nextPosition++] = (ppsLen) & 0xFF; // Copy SPS data memcpy(&rtmpPacket->m_body[nextPosition], pps, ppsLen); // Set RTMP package type and video type data rtmpPacket->m_packetType = RTMP_PACKET_TYPE_VIDEO; // Set RTMP package length rtmpPacket->m_nBodySize = rtmpPackagesize; // Assign RTMP channels, assign at will rtmpPacket->m_nChannel = 10; // Set the video time stamp. If it is SPP PPS data, there is no time stamp rtmpPacket->m_nTimeStamp = 0; // Set the absolute time, and assign 0 to SPS PPS rtmpPacket->m_hasAbsTimestamp = 0; // Set the header type, and set one at will rtmpPacket->m_headerType = RTMP_PACKET_SIZE_MEDIUM; }