The relevant design of OpenMax is very excellent. This paper mainly analyzes and summarizes the IL layer of OpenMax from the design point of view.
For the basic concepts of OpenMax and the basic introduction of IL layer, please refer to another personal summary about encoding and decoding abstraction layer OpenMax: https://blog.csdn.net/runafterhit/article/details/119961868
1, Design concept and feature points of OpenMax
Let's briefly sort out the design concepts and features of OpenMax to analyze the subsequent design points. (this part has been summarized in the previous blog post)
Design philosophy - abstraction / portability / asynchronous processing / component composition of media framework
In order to improve the portability of multimedia and connect with the current device media diversity schemes, openmax IL is abstracted to a higher level compared with these schemes, which usually means a multimedia framework. The interface of openmax IL layer comes from the media framework level. It is easier to integrate new decoders in design and implementation. Other interfaces, such as file processing, can also be added conveniently, taking into account various scalability. At the same time, highly asynchronous communication mode is used in the design, which can be processed through one or more execution threads or special hardware IP (hardware acceleration). The way of multi component link processing also gives the architecture greater flexibility and efficiency.
Design features - componentized API / easy to add decoder / easy to expand / support dynamic link / configurable
The core API interface design based on component componentization maintains flexibility.
It can easily add the ability to integrate new decoder codec.
At the same time, it provides Khronos Group and suppliers with good scalability in the fields of audio, video and image;
It can be implemented as a static library or a dynamic link library.
Provides key features and configuration options for parent platforms such as multimedia frameworks.
Maintain the simplicity of communication between client and decoder codec and decoder codec;
2, Design point analysis of OpenMax
Sort out the real design points of openmax and summarize them according to personal ideas as follows:
2.1 [compatibility] version compatibility design - component version number \ pointer function \ input parameter void pointer
(1) Component version number: because the component of OMX can be loaded dynamically, the. so of each component is implemented by the chip vendor. They may have different versions or alternative schemes. Therefore, OMX provides OMX_ The getcomponentversion interface is used to query the version number of components. The caller needs to judge the use policy according to the version number (usually, the framework may only adapt to one component and only be used for verification. When there are multiple versions of components, it needs to refer to the running policy). The version number is generally composed of a set of 8bit fields in the static compilation stage, as follows:
// The version number of omx defines the data structure
typedef union OMX_VERSIONTYPE {
struct {
OMX_U8 nVersionMajor; /**< Major version accessor element */
OMX_U8 nVersionMinor; /**< Minor version accessor element */
OMX_U8 nRevision; /**< Revision version accessor element */
OMX_U8 nStep; /**< Step version accessor element */
} s;
OMX_U32 nVersion; /**< 32 bit value to make accessing the version easily done in a single word size copy/compare operation */
} OMX_VERSIONTYPE;
// Typical omx version number examples
#define OMX_VERSION_MAJOR 1
#define OMX_VERSION_MINOR 2
#define OMX_VERSION_REVISION 0
#define OMX_VERSION_STEP 0
#define OMX_VERSION ((OMX_VERSION_STEP<<24) | (OMX_VERSION_REVISION<<16) | (OMX_VERSION_MINOR<<8) | OMX_VERSION_MAJOR)
(2) Pointer function: in order to unify the external interface and encapsulate the differences between different chip manufacturers, the omx interface is basically called through the function pointer, as shown in setparameter. The user uses the core API macro function OMX_SetParameter actually calls the setparameter function pointer of the component. The setparameter function pointer is usually used when the component initializes omx_ When componentinit is registered, it is implemented as the corresponding package of its own chip hard solution scheme, such as xxx_SetParameter.
// core interface
#define OMX_SetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure) \
((OMX_COMPONENTTYPE*)hComponent)->SetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure)
// Component interface
OMX_ERRORTYPE (*SetParameter)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_INDEXTYPE nIndex,
OMX_IN OMX_PTR pComponentParameterStructure);
// Register function pointers when initializing components
OMX_ERRORTYPE OMX_ComponentInit(OMX_HANDLETYPE phandle){
// slightly
phandle->SetParameter = xxx_SetParameter; // Docking with real interface implementation
// slightly
}
(3) Input parameter void pointer: in order to ensure the compatibility of various component implementations, omx input parameters are basically passed through null pointers, and parameter data structure types are described in combination with parameter type index or input parameter size; For example, in the setParameter function above, the core interface is a macro function, the interface of the component component actually called, and the parameters are entered by enumerating OMX_INDEXTYPE describes the type of parameter. Parameter data is passed in through void * type pointers. In this way, various types of configurations (including extensible private data structures) can be set through a setParam interface.
typedef enum OMX_INDEXTYPE {
OMX_IndexComponentStartUnused = 0x01000000,
OMX_IndexParamPriorityMgmt, /**< reference: OMX_PRIORITYMGMTTYPE */
OMX_IndexParamAudioInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamImageInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamVideoInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamOtherInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamNumAvailableStreams, /**< reference: OMX_PARAM_U32TYPE */
// slightly
} OMX_INDEXTYPE;
typedef void* OMX_PTR;
2.2 [extensibility] new component design - new components in dynamic library
(1) Dynamic library loading design: in omx design, each decoder component is made into a separate. So to be realized through dynamic loading. Via OMX_GetHandle passes in the decoder name, creates a decoder instance, loads so and obtains the corresponding omx in this process_ Componentinit initializes the symbol name, registers the corresponding pointer function in so, and provides subsequent access. The key parts are as follows:
// OMX_GetHandle creates the key part of the decoder instance
OMX_API OMX_ERRORTYPE OMX_APIENTRY OMX_GetHandle(
OMX_OUT OMX_HANDLETYPE* pHandle,
OMX_IN OMX_STRING cComponentName,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_CALLBACKTYPE* pCallBacks) {
// slightly
dlhandle = dlopen (libname, RTLD_LAZY | RTLD_GLOBAL); // Get the handle of the library through the dlopen library name
// libname is usually generated according to fixed rules with cComponentName
entry = dlsym (dlhandle, "OMX_ComponentInit")); // Get the symbolic address of the initialization function through the access handle of the library
ret = entry(*pHandle); // Via OMX_ The symbol of componentinit is called to the so library for real decoder initialization
}
// OMX_ The key implementation of componentinit (in the corresponding so Library)
OMX_ERRORTYPE OMX_ComponentInit (OMX_HANDLETYPE handle) {
OMX_COMPONENTTYPE *phdl = (OMX_COMPONENTTYPE *) handle; // Force handle to instance standard context
p_hdl->SendCommand = SendCommand; // Register component callback for use
p_hdl->SetParameter = SetParameter;
p_hdl->SetConfig = SetConfig;
p_hdl->EmptyThisBuffer = EmptyThisBuffer;
// slightly
}
Through this library loading method, the implementation of the decoder is encapsulated into one so and isolated, which can be loaded and used when needed;
2.3 [extensibility] new component new private function design - extended parameter setting type enumeration
(1) Extended parameter setting type enumeration: omx's common calling processes define corresponding pointer functions, such as SendCommand, SetConfig configuration parameters, EmptyThisBuffer and other buffer process calls. We have learned about the basic usage of SetParameter. cmd type enumeration indicates what parameters are set, and void * passes parameter data. When a new setting needs to be extended, the new parameter type is extended. The extended function type is obtained by passing in the function name through the GetExtensionIndex function pointer of the component.
// When defining a standard cmd type, many middle bits have been reserved, which are defined by segments
typedef enum OMX_INDEXTYPE {
OMX_IndexComponentStartUnused = 0x01000000,
OMX_IndexParamPriorityMgmt, /**< reference: OMX_PRIORITYMGMTTYPE */
OMX_IndexParamAudioInit, /**< reference: OMX_PORT_PARAM_TYPE */
// slightly
OMX_IndexPortStartUnused = 0x02000000,
OMX_IndexParamPortDefinition, /**< reference: OMX_PARAM_PORTDEFINITIONTYPE */
OMX_IndexParamCompBufferSupplier, /**< reference: OMX_PARAM_BUFFERSUPPLIERTYPE */
OMX_IndexReservedStartUnused = 0x03000000,
// slightly
/* Audio parameters and configurations */
OMX_IndexAudioStartUnused = 0x04000000,
OMX_IndexParamAudioPortFormat, /**< reference: OMX_AUDIO_PARAM_PORTFORMATTYPE */
// slightly
OMX_IndexMax = 0x7FFFFFFF
} OMX_INDEXTYPE;
// Through the api interface of the core, call the function pointer GetExtensionIndex of the component to query the extension command
#define OMX_GetExtensionIndex( \
hComponent, \
cParameterName, \
pIndexType) \
((OMX_COMPONENTTYPE*)hComponent)->GetExtensionIndex( \
hComponent, \
cParameterName, \
pIndexType)
// The query extension command function pointer of the component passes in the function method name or method enumeration to set the use
OMX_ERRORTYPE (*GetExtensionIndex)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_STRING cParameterName,
OMX_OUT OMX_INDEXTYPE* pIndexType);
2.4 [performance] asynchronous processing performance design - configure asynchronous notification / buffer rotation callback notification
Performance design is an important part of multimedia framework, which mainly includes two aspects
Asynchronous processing mechanism based on event and callback
(1) Configure asynchronous notification: whether the interface is a synchronous blocking function or an asynchronous non blocking function. A large number of types of interfaces in the omx framework suggest that it is a design level asynchronous interface, such as parameter setting and port parameter adjustment taking effect. The asynchronous event callback is used to notify the upper layer when the action is completed.
(2) Buffer rotation callback notification: the most original is to send the input / output buffer to the decoder, and then poll the acquiree output dequeue input for acquisition. This method is based on circular traversal, occupies scheduling performance and is not timely enough. omx makes the upper layer fetch frames more timely through EmptyBufferDone and EmptyBufferDone events, and cyclic scheduling is not required.
// In OMX_ When gethandle creates a decoder instance, the event notification callback is included in the set callback.
OMX_API OMX_ERRORTYPE OMX_APIENTRY OMX_GetHandle(
OMX_OUT OMX_HANDLETYPE* pHandle,
OMX_IN OMX_STRING cComponentName,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_CALLBACKTYPE* pCallBacks); // Register callback type
// The registered callback types are as follows, including events, input frame usage completion, and output frame filling completion
typedef struct OMX_CALLBACKTYPE {
OMX_ERRORTYPE (*EventHandler)( // Event notification callback
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_EVENTTYPE eEvent,
OMX_IN OMX_U32 nData1,
OMX_IN OMX_U32 nData2,
OMX_IN OMX_PTR pEventData);
OMX_ERRORTYPE (*EmptyBufferDone)( // The input frame uses the completion event callback to fill in the incoming component
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
OMX_ERRORTYPE (*FillBufferDone)( //The input frame completes the event callback, and the output can be used
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
} OMX_CALLBACKTYPE;
Note: it is stated in the omx design document that the event callback processing function must meet the thread safety design. Callbacks may occur at the same time (the same callback is concurrent, or different callbacks are concurrent). Pay attention to the safety protection design of the critical area during concurrent execution, such as using semaphore and other mechanisms; The bottom layer cannot execute callbacks (such as hardware interrupts) in non threads to avoid blocking in the upper layer callbacks and causing system exceptions; Avoid time-consuming operations in callbacks. In actual project implementation, independent threads are often used for event notification at the bottom, while in the upper layer, when callbacks may take more time, separate threads need to be used to receive event execution instead of being implemented directly in callbacks.
2.5 [codec field] establish the buffer automatic rotation path design point in the tunnel mode between components
The previous introduction of buffer rotation callback notification explains the asynchronous notification design mechanism of buffer rotation. Through this design, the series scheduling between the underlying components can be realized without the framework or users participating in buffer rotation. The components can directly manage buffer rotation through callback events, which is also called the tunnel communication mode between components. As shown in the figure below, source Component and Host Component are common calling methods other than tunnel. The framework needs to trigger rotation according to EmptyBufferDone and EmptyBufferDone events, and the latter components are tunnel mode.
Example of tunnel mode: camera acquisition component mjpeg decoding component render display component is bound to tunnel mode. When camera collects a frame, the bottom driver triggers filldone event to send the frame EmptyThisBuffer to mjpeg decoding component, and mjpeg decoding component decodes and sends the frame EmptyThisBuffer to render display module for rendering display, regardless of whether the frame is sent or not, Neither user nor framework should participate in buffer rotation scheduling.
2.6 [codec field] component basic abstract design - handle encapsulation / port abstraction / internal and external buffer /state
(1) Handle encapsulation: the user or framework accesses the component through handle and calls the core api to pass in the handle, which only needs to include omx_ The header file of core. H is OK. This handle type is OMX_HANDLETYPE is actually a void * type pointer pointing to the context of the component. The real definition of context is in omx_ The header file of component. H defines the real data structure OMX_COMPONENTTYPE, which is invisible to the user, is visible only when the core implementation of omx is visible, so as to encapsulate the internal details of the component. Simultaneous OMX_COMPONENTTYPE also contains the pComponentPrivate pointer to the real internal private context of the decoder, which is similar to the private driving access to the file handle. This context is only visible in the internal access of the component, and omx core is not visible, generally in omx_ During componentinit, malloc private context is attached to pComponentPrivate and used internally in subsequent component access;
typedef void* OMX_HANDLETYPE; // handle seen by user and framework
typedef struct OMX_COMPONENTTYPE { // omx core implementation converts handle to true context implementation
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_PTR pComponentPrivate; // omx component component private data structure context access
OMX_PTR pApplicationPrivate;
OMX_ERRORTYPE (*GetComponentVersion)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_OUT OMX_STRING pComponentName,
OMX_OUT OMX_VERSIONTYPE* pComponentVersion,
OMX_OUT OMX_VERSIONTYPE* pSpecVersion,
OMX_OUT OMX_UUIDTYPE* pComponentUUID);
OMX_ERRORTYPE (*SendCommand)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_COMMANDTYPE Cmd,
OMX_IN OMX_U32 nParam1,
OMX_IN OMX_PTR pCmdData);
// slightly
} OMX_COMPONENTTYPE;
(2) Port abstraction: port is the abstraction of data port, which is divided into input in port and output out port. Generally, common components are input and output one by one. If it is a source component, it may only have output, and the last level, such as render, may only have input. Some components may have multiple ports. For example, the output of processing components that support one in many out may have multiple ports;
(3) Internal and external buffer mode: the buffer mode indicates whether the memory of a port is applied internally or sent externally. This is a common decoder usage. Using an external buffer can make it easier for the upper layer to use and manage.
(4) State machine: each component has a state machine to manage internal state switching. Each component is first considered unloaded, loaded by the CoreAPI, and then transformed with other states in the communication process. When Invalid data is encountered, the component may enter the Invalid invalid state. The behavior and operation of components in different states are different and limited. State switching calls OMX through SendCommand_ The commandstateset type is set by the command.
3, Thoughts on Multimedia Framework
Why does codec need framework support so much - Codec Requirements / hard decoding private / user isolation
(1) With the development of audio and video technology, the performance requirements of encoding and decoding are increasing. fhd is from 4k to 8k, frame rate is from 24hz to 60hz to 120hz, and hdr and other technologies to improve the display restoration effect appear. Especially with the development of mobile devices such as mobile phones, the previous soft decoding methods are difficult to ensure stable performance, and soft decoding has very high requirements for cpu performance.
(2) Therefore, various equipment manufacturers are accelerating through hard decoding. The hard decoding schemes are often very privatized. There are a variety of decoding protocol implementation schemes. Many ambiguous protocols rely on a large number of problem processing iterations, which belong to key technical assets and form technical barriers. They are usually released through closed sources, For example, running on the mcu rather than on the kernel device to avoid kernel license infection, the playing methods are naturally very different. And different decoders on the same device may come from different teams, or even often different companies.
(3) Users can't see these differences naturally. Therefore, users usually face multimedia framework or even higher multiplication packaging. The framework connects various soft decoding and hard decoder schemes downward, and these decoders are not independent. They often need to be combined, and they also have high requirements for performance (software scheduling and asynchronous processing), If there is no set of standards to connect, it will be extremely complex.
(4) Furthermore, different platforms have their own framework. If each chip manufacturer adapts it once, it will be extremely difficult. Strictly speaking, openmax should be the packaging layer between the docking framework and the hard decoding scheme, not a framework. For example, ffmpeg, Gstreamer and Android mediaCodec can dock with openmax, Hardware manufacturers only need to connect with openmax.
How to build a good general multimedia framework (to be updated)
What good designs can the private multimedia channel draw from OpenMax (to be updated)
reference resources
Detailed documents of IL layer on the official website: https://www.khronos.org/files/openmax_il_spec_1_0.pdf
Introduction to encoding and decoding abstraction layer OpenMax: https://blog.csdn.net/runafterhit/article/details/119961868
omx header file: http://androidxref.com/9.0.0_r3/xref/frameworks/native/headers/media_plugin/media/openmax/