How does Python interact with C code in microphoton

1. Python code and C code character association

This section describes how Python characters are associated with C characters (such as module name, class name, variable name, function name, etc.) during module expansion? Take the ADC class name symbol as an example to illustrate the following association process:

C code character ADC

STATIC const mp_rom_map_elem_t driver_locals_dict_table[] = {                                       
    {MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_driver)},                               
    {MP_OBJ_NEW_QSTR(MP_QSTR_ADC), MP_ROM_PTR(&driver_adc_type)},                                   
   ......
};

Python code character ADC

from driver import ADC

Character association method

By executing the script command: (script path: components/py_engine/engine/genhdr/gen_qstr.py)

python gen_qstr.py ADC

Generate a unique character mapping relationship:

QDEF(MP_QSTR_ADC, (const byte*)"\x63\x03" "ADC")

Store the results in the file components / py_ In engine / adapter / Haas / genhdr / qstrdefs.generated.h, the character mapping is completed. During code execution, the python engine maps the associated characters through the information of qstrdefs.generated.h.

2. Python code calls C code

When the Python engine runs, it will look for the macro definition Microsoft_ PORT_ BUILTIN_ Modules, which is used as the entry of component extension, netmgr_module and driver_module is the name of the extension module defined above.

Code path: components/py_engine/adapter/haas/mpconfigport.h

#define MICROPY_PORT_BUILTIN_MODULES \                                                                                                                                                                   
        {MP_ROM_QSTR(MP_QSTR_netmgr), MP_ROM_PTR(&netmgr_module)}, \                                
        {MP_ROM_QSTR(MP_QSTR_driver), MP_ROM_PTR(&driver_module)},

In this way, the C extension component is associated with the python engine. Python applications can import netmgr module through import netmgr, import ADC class from module driver through from driver import ADC, and call the functions provided by the module / class.

3. C code calls Python code

In embedded development, the event notification of most peripheral interfaces (such as Timer, UART, GPIO, etc.) is realized through callback function. The status notification of some modules is also realized through callback, such as the network status change notification function. In the conventional development based on c voice, ISR (interrupt callback function) works in the context of system process / thread, and the callback notification mechanism is easy to control. However, python applications work in the context of Python virtual machine processes, and interrupt callback functions occur in the context of C underlying processes. C processes and python virtual machine processes are isolated from each other, so the path for C language to directly call Python code is impassable.

Figure 1 process of C and Python ISR

Micropathon provides two ways to realize the communication from C bottom process to Python virtual machine process, and realizes the function of C language notifying Python application layer. Next, this section takes the Timer module as an example to analyze the principles of the two callback mechanisms in detail, so as to facilitate developers to expand their own modules to the micro Python system and jointly enrich and develop the python light application ecology.

First, create and initialize the ISR thread virtualization environment so that the ISR thread can obtain the same context as the Python virtual machine process.

//1 get and save the current virtual machine thread state 
void *old_state = mp_thread_get_state();
//2 allocate and set the status information of ISR thread
mp_state_thread_t ts;
mp_thread_set_state(&ts);
//3 initialize the stack pointer of the new virtual machine thread of ISR, + 1 indicates that ts information needs to be included in the following pointer scanning
mp_stack_set_top(&ts + 1);
//4 set the stack size of the new thread virtual machine according to the stack size of the ISR thread. The stack size depends on the ISR thread stack. This value will change in different modules
//  Change. (pain point 1)
mp_stack_set_limit(1024);
//5 pass the local and global status information of the current virtual machine thread to the newly created thread
mp_locals_set(mp_locals_get());
mp_globals_set(mp_globals_get());
//6 prohibit thread scheduling of virtual machine to prevent virtual machine from switching to other micro Python threads
mp_sched_lock();
//7 masked memory allocation
gc_lock();
//8 execute micro Python APIs callback and complete the callback from C bottom layer to Python application layer (pain point 2)
mp_call_function_1_protected(callback, MP_OBJ_FROM_PTR(arg));
//9 enable memory allocation
gc_unlock();
//10 enable virtual machine thread scheduling
mp_sched_unlock();
//11 restore the virtual machine thread state to the state saved in the first step 
mp_thread_set_state(old_state);

Calling the callback function of the Python thread in the ISR of C language takes 11 steps to complete, and there are two pain points:

In step 4, you need to evaluate the stack size of ISR threads to set the stack of the new thread virtual environment. It is not easy to evaluate the thread stack size.
In step 8, you need to determine the function call according to the number of callback parameters of ISR. When there are more callback parameters, you need to convert multiple parameters into dictionary variables for callback. Currently, micropathon provides callback function definitions for two variables, as shown below:

mp_obj_t mp_call_function_1_protected(mp_obj_t fun, mp_obj_t arg);
mp_obj_t mp_call_function_2_protected(mp_obj_t fun, mp_obj_t arg1, mp_obj_t arg2);

It can be seen that although the above method can realize the callback from ISR to Python application layer, it takes 11 steps to complete, and the stack size of the new thread is not easy to evaluate. Can there be another mechanism? Micro Python provides a second callback mechanism: looper handler mode. In this mode, the ISR thread injects the callback function handle passed from the python application layer into the virtual machine environment and notifies the python main thread to query and call the callback function in the python main thread, so there is no need to create a new virtual thread.

The micro Python looper handler mode provides mp_sched_schedule function, which allows ISR to register callback functions into the virtual machine environment. When the python main thread parses and executes the script code, it will check the scheduling status of the virtual machine, and then decide whether to execute the callback function. Here is MP_ sched_ Implementation of schedule function:

bool MICROPY_WRAP_MP_SCHED_SCHEDULE(mp_sched_schedule)(mp_obj_t function, mp_obj_t arg) {
    mp_uint_t atomic_state = MICROPY_BEGIN_ATOMIC_SECTION();
    bool ret;
    //1 check whether the scheduling queue is full. The callback function can be injected only when the queue is not full
    if (!mp_sched_full()) {
        if (MP_STATE_VM(sched_state) == MP_SCHED_IDLE) {
            //2. Set the scheduling status to facilitate the query and execution of the main thread of the subsequent virtual machine
            MP_STATE_VM(sched_state) = MP_SCHED_PENDING;
        }
        //3 increase the index of the scheduling queue and inject the callback function
        uint8_t iput = IDX_MASK(MP_STATE_VM(sched_idx) + MP_STATE_VM(sched_len)++);
        MP_STATE_VM(sched_queue)[iput].func = function;
        MP_STATE_VM(sched_queue)[iput].arg = arg;
         //4. The callback injection succeeds and returns true
        ret = true;
    } else {
        //5. The scheduling queue is full, and false is returned
        ret = false;
    }
    MICROPY_END_ATOMIC_SECTION(atomic_state);
    return ret;
}

Micro Python uses the precompiled parameter Microsoft_ SCHEDULER_ Depth sets the depth of the scheduling queue, which is 4 by default.

// Maximum number of entries in the scheduler
#ifndef MICROPY_SCHEDULER_DEPTH
#define MICROPY_SCHEDULER_DEPTH (4)
#endif

HaaS light applications encapsulate Microsoft_ EVENT_ POLL_ Hook macro definition: call the macro when the callback function needs to be executed immediately in REPL (Interactive interpreter) mode or other situations to trigger the virtual machine thread to call mp_handle_pending(true) to complete mp_ handle_ pending_ Call the tail function, and finally realize the callback injected into the scheduling queue function. The code implementation of this macro definition is as follows:

#define MICROPY_EVENT_POLL_HOOK \
    do { \
        extern void mp_handle_pending(bool); \
        mp_handle_pending(true); \
        MICROPY_PY_USOCKET_EVENTS_HANDLER \
        MP_THREAD_GIL_EXIT(); \
        MP_THREAD_GIL_ENTER(); \
    } while (0);
// A variant of this is inlined in the VM at the pending exception check
void mp_handle_pending(bool raise_exc) {
    if (MP_STATE_VM(sched_state) == MP_SCHED_PENDING) {    // Check whether the scheduling status is set to MP_SCHED_PENDING status
        mp_uint_t atomic_state = MICROPY_BEGIN_ATOMIC_SECTION();
        // Re-check state is still pending now that we're in the atomic section.
        if (MP_STATE_VM(sched_state) == MP_SCHED_PENDING) {
            mp_obj_t obj = MP_STATE_VM(mp_pending_exception);
            if (obj != MP_OBJ_NULL) {
                ...
            }
            mp_handle_pending_tail(atomic_state);    // Call callback function
        } else {
            MICROPY_END_ATOMIC_SECTION(atomic_state);
        }
    }
}

MP can be called directly in lexer (lexical analyzer) mode_ handle_ pending_ The tail function implements callback triggering, which is not analyzed in detail in this section, MP_ handle_ pending_ The code implementation of Tai is as follows.

// This function should only be called by mp_handle_pending,
// or by the VM's inlined version of that function.
void mp_handle_pending_tail(mp_uint_t atomic_state) {
    MP_STATE_VM(sched_state) = MP_SCHED_LOCKED;
    if (!mp_sched_empty()) {
        mp_sched_item_t item = MP_STATE_VM(sched_queue)[MP_STATE_VM(sched_idx)];
        MP_STATE_VM(sched_idx) = IDX_MASK(MP_STATE_VM(sched_idx) + 1);
        --MP_STATE_VM(sched_len);
        MICROPY_END_ATOMIC_SECTION(atomic_state);
        mp_call_function_1_protected(item.func, item.arg);
    } else {
        MICROPY_END_ATOMIC_SECTION(atomic_state);
    }
    mp_sched_unlock();
}

From the above introduction, we can see that the second method only needs to call a function to realize the injection of callback function, which is very convenient for developers. The sample code of ISR function in Timer module is posted below for readers' reference.

STATIC void driver_timer_isr(void *self_in) {
    driver_timer_obj_t *self = (driver_timer_obj_t*)self_in;
    if (self->callback != mp_const_none) {
        bool ret = mp_sched_schedule(self->callback, MP_OBJ_FROM_PTR(self));
        if(ret == false) {
            printf("[utility]: schedule queue is full !!!!\r\n");
        }
    }
}

4. Summary

The HaaS team encapsulates the rich software and hardware building block capabilities of the underlying HaaS into a python library through component expansion for direct use by Python application layer code, which greatly improves the production efficiency of Python light applications.