ijkplayer is more popular now, because of working relationship, contact him, now do a simple analysis record. I skip the java layer code and go to the c layer directly, because most of the work is done through jni calls to the c layer, the content of the java layer is not the main function.
Let's look at the clues first. Look directly at the ijkplayer_jni.c file, under ijkmedia. All C functions and java function mapping relationships are here. The _prepareAsync method invoked by the upper layer of java is the native method, where it is mapped to IjkMediaPlayer_prepareAsync. After a series of calls, you will go to the ijkmp_prepare_async_l method of ijkplayer.c. Here you can see:
ijkmp_change_state_l(mp, MP_STATE_ASYNC_PREPARING); msg_queue_start(&mp->ffplayer->msg_queue); // released in msg_loop ijkmp_inc_ref(mp); mp->msg_thread = SDL_CreateThreadEx(&mp->_msg_thread, ijkmp_msg_loop, mp, "ff_msg_loop"); // msg_thread is detached inside msg_loop // TODO: 9 release weak_thiz if pthread_create() failed; int retval = ffp_prepare_async_l(mp->ffplayer, mp->data_source); if (retval < 0) { ijkmp_change_state_l(mp, MP_STATE_ERROR); return retval; }
This shows that a queue is started, a loop message thread is started, and a key function ffp_prepare_async_l is taken. The so-called startup queue is actually the allocation space initialization, and then crammed into a private message body. In the latter loop thread, the real executor is message_loop of ijkplayer_jni.c. Let's leave him alone for the time being and understand it as a separate thread to handle message dispatch. Focus on the following ffp_prepare_async_l, which is the stage to be ready to play:
int ffp_prepare_async_l(FFPlayer *ffp, const char *file_name) { assert(ffp); assert(!ffp->is); assert(file_name); if (av_stristart(file_name, "rtmp", NULL) || av_stristart(file_name, "rtsp", NULL)) { // There is total different meaning for 'timeout' option in rtmp av_log(ffp, AV_LOG_WARNING, "remove 'timeout' option for rtmp.\n"); av_dict_set(&ffp->format_opts, "timeout", NULL, 0); } /* there is a length limit in avformat */ if (strlen(file_name) + 1 > 1024) { av_log(ffp, AV_LOG_ERROR, "%s too long url\n", __func__); if (avio_find_protocol_name("ijklongurl:")) { av_dict_set(&ffp->format_opts, "ijklongurl-url", file_name, 0); file_name = "ijklongurl:"; } } av_log(NULL, AV_LOG_INFO, "===== versions =====\n"); ffp_show_version_str(ffp, "ijkplayer", ijk_version_info()); ffp_show_version_str(ffp, "FFmpeg", av_version_info()); ffp_show_version_int(ffp, "libavutil", avutil_version()); ffp_show_version_int(ffp, "libavcodec", avcodec_version()); ffp_show_version_int(ffp, "libavformat", avformat_version()); ffp_show_version_int(ffp, "libswscale", swscale_version()); ffp_show_version_int(ffp, "libswresample", swresample_version()); av_log(NULL, AV_LOG_INFO, "===== options =====\n"); ffp_show_dict(ffp, "player-opts", ffp->player_opts); ffp_show_dict(ffp, "format-opts", ffp->format_opts); ffp_show_dict(ffp, "codec-opts ", ffp->codec_opts); ffp_show_dict(ffp, "sws-opts ", ffp->sws_dict); ffp_show_dict(ffp, "swr-opts ", ffp->swr_opts); av_log(NULL, AV_LOG_INFO, "===================\n"); av_opt_set_dict(ffp, &ffp->player_opts); if (!ffp->aout) { ffp->aout = ffpipeline_open_audio_output(ffp->pipeline, ffp); if (!ffp->aout) return -1; } #if CONFIG_AVFILTER if (ffp->vfilter0) { GROW_ARRAY(ffp->vfilters_list, ffp->nb_vfilters); ffp->vfilters_list[ffp->nb_vfilters - 1] = ffp->vfilter0; } #endif VideoState *is = stream_open(ffp, file_name, NULL); if (!is) { av_log(NULL, AV_LOG_WARNING, "ffp_prepare_async_l: stream_open failed OOM"); return EIJK_OUT_OF_MEMORY; } ffp->is = is; ffp->input_filename = av_strdup(file_name); return 0; }
Before judging the live video streaming protocol rtmp or rtsp, and then basically output a pair of information, the real core function is stream_open. This is to open the video stream according to the address. The code is as follows:
static VideoState *stream_open(FFPlayer *ffp, const char *filename, AVInputFormat *iformat) { assert(!ffp->is); VideoState *is; is = av_mallocz(sizeof(VideoState)); if (!is) return NULL; is->filename = av_strdup(filename); if (!is->filename) goto fail; is->iformat = iformat; is->ytop = 0; is->xleft = 0; /* start video display */ if (frame_queue_init(&is->pictq, &is->videoq, ffp->pictq_size, 1) < 0) goto fail; if (frame_queue_init(&is->subpq, &is->subtitleq, SUBPICTURE_QUEUE_SIZE, 0) < 0) goto fail; if (frame_queue_init(&is->sampq, &is->audioq, SAMPLE_QUEUE_SIZE, 1) < 0) goto fail; if (packet_queue_init(&is->videoq) < 0 || packet_queue_init(&is->audioq) < 0 || packet_queue_init(&is->subtitleq) < 0) goto fail; if (!(is->continue_read_thread = SDL_CreateCond())) { av_log(NULL, AV_LOG_FATAL, "SDL_CreateCond(): %s\n", SDL_GetError()); goto fail; } init_clock(&is->vidclk, &is->videoq.serial); init_clock(&is->audclk, &is->audioq.serial); init_clock(&is->extclk, &is->extclk.serial); is->audio_clock_serial = -1; is->audio_volume = SDL_MIX_MAXVOLUME; is->muted = 0; is->av_sync_type = ffp->av_sync_type; is->play_mutex = SDL_CreateMutex(); ffp->is = is; is->pause_req = !ffp->start_on_prepared; is->video_refresh_tid = SDL_CreateThreadEx(&is->_video_refresh_tid, video_refresh_thread, ffp, "ff_vout"); if (!is->video_refresh_tid) { av_freep(&ffp->is); return NULL; } is->read_tid = SDL_CreateThreadEx(&is->_read_tid, read_thread, ffp, "ff_read"); if (!is->read_tid) { av_log(NULL, AV_LOG_FATAL, "SDL_CreateThread(): %s\n", SDL_GetError()); fail: is->abort_request = true; if (is->video_refresh_tid) SDL_WaitThread(is->video_refresh_tid, NULL); stream_close(ffp); return NULL; } return is; }
Here you can see three queues initialized, video, audio and subtitles. These queues are divided into two queues through frame_queue_init, original and decoded. Looking down, there are two threads: video_refresh_thread s and read_thread s. Literally, they should be output refresh video and read threads. Let's start with reading threads. The whole code is very long. Notice the ic variables here. It's an AVFormatContext type, where streaming information and data are stored.
...... ic = avformat_alloc_context(); ...... err = avformat_open_input(&ic, is->filename, is->iformat, &ffp->format_opts); ...... err = avformat_find_stream_info(ic, opts); ...... for (i = 0; i < ic->nb_streams; i++) { AVStream *st = ic->streams[i]; enum AVMediaType type = st->codecpar->codec_type; st->discard = AVDISCARD_ALL; if (type >= 0 && ffp->wanted_stream_spec[type] && st_index[type] == -1) if (avformat_match_stream_specifier(ic, st, ffp->wanted_stream_spec[type]) > 0) st_index[type] = i; // choose first h264 if (type == AVMEDIA_TYPE_VIDEO) { enum AVCodecID codec_id = st->codecpar->codec_id; video_stream_count++; if (codec_id == AV_CODEC_ID_H264) { h264_stream_count++; if (first_h264_stream < 0) first_h264_stream = i; } } } ...... /* open the streams */ if (st_index[AVMEDIA_TYPE_AUDIO] >= 0) { stream_component_open(ffp, st_index[AVMEDIA_TYPE_AUDIO]); } ret = -1; if (st_index[AVMEDIA_TYPE_VIDEO] >= 0) { ret = stream_component_open(ffp, st_index[AVMEDIA_TYPE_VIDEO]); } if (is->show_mode == SHOW_MODE_NONE) is->show_mode = ret >= 0 ? SHOW_MODE_VIDEO : SHOW_MODE_RDFT; if (st_index[AVMEDIA_TYPE_SUBTITLE] >= 0) { stream_component_open(ffp, st_index[AVMEDIA_TYPE_SUBTITLE]); } ...... /* offset should be seeked*/ if (ffp->seek_at_start > 0) { ffp_seek_to_l(ffp, ffp->seek_at_start); } for (;;) { ...... ret = av_read_frame(ic, pkt); ...... /* check if packet is in play range specified by user, then queue, otherwise discard */ stream_start_time = ic->streams[pkt->stream_index]->start_time; pkt_ts = pkt->pts == AV_NOPTS_VALUE ? pkt->dts : pkt->pts; pkt_in_play_range = ffp->duration == AV_NOPTS_VALUE || (pkt_ts - (stream_start_time != AV_NOPTS_VALUE ? stream_start_time : 0)) * av_q2d(ic->streams[pkt->stream_index]->time_base) - (double)(ffp->start_time != AV_NOPTS_VALUE ? ffp->start_time : 0) / 1000000 <= ((double)ffp->duration / 1000000); ......@@@ if (pkt->stream_index == is->audio_stream && pkt_in_play_range) { packet_queue_put(&is->audioq, pkt); } else if (pkt->stream_index == is->video_stream && pkt_in_play_range && !(is->video_st && (is->video_st->disposition & AV_DISPOSITION_ATTACHED_PIC))) { packet_queue_put(&is->videoq, pkt); } else if (pkt->stream_index == is->subtitle_stream && pkt_in_play_range) { packet_queue_put(&is->subtitleq, pkt); } else { av_packet_unref(pkt); } } ......
avformat_alloc_context is the allocation space initialization. Then fill in ic in avformat_open_input, which reads the information of network packet, decides the format and so on. The function avformat_find_stream_info doesn't look very clear inside, but it feels like it's looking for a decoder. Let's analyze this later. Let's look at the big process first. This for loop takes out each frame, then judges the type, classifies the audio and video subtitles, and sets the number separately. Then we come to the core function, stream_component_open, which reads and decodes streams according to different types of frames. There was a question here. Would it be more efficient to process frame data before decoding, such as optimizing frame loss processing? No, if it is not decoded, it is impossible to distinguish which frame in gop is the key frame i frame, which brings great inconvenience to the subsequent processing (hereinafter noted). Real read in av_read_frame, read the decoded data, read into pkt. This process is performed inside a for infinite loop, after seek's processing. This for will be missed carelessly... So what data does this av_read_frame read? Looking at the structure AVPacket, the explanation is quite clear. It's a compressed data frame. Whether the key frame has been marked in flags or not. Whatever is done before and after this cycle, it's a step to read data frames. Here is a large section of the treatment of errors, temporarily skipped. The following section checks whether the packet is in the user-specified playback range and queues according to the timestamp, otherwise discards the process. Starting with stream_start_time=..., what i understand is that after calculating the timestamp of the current data frame, i can calculate the start time of playback to the current time, and then see if the timestamp is in this range. put the range into the queue, or discard it. Insert a sentence here, you can add frame loss processing in live broadcasting at @@here. Details can be found online. Finally, buffer checking was done before the cycle. Then there's error handling, and that's how the whole process works.
Every skipping place here is actually quite complicated. I'll pick up some key points to do the next analysis. I have just come into contact with this kind of technology and only take notes.