Python asynchronous programming

preface

Now it's Python 3.5 and it has entered the asynchronous era

Because of the existence of GIL (global lock), python can not play the advantage of multi-core, and its performance has been criticized. However, in IO intensive network programming, asynchronous processing is hundreds of times more efficient than synchronous processing, making up for Python's performance shortcomings

• In the era of Python 3.0, the asynchronous network module in the standard library: Select (very low level)

• In the era of Python 3.0, the third-party asynchronous network library: Tornado

• In the era of Python 3.4, asyncio: supports TCP and subprocesses. It has built-in support for asynchronous IO.

Existing python asynchronous framework

• tornado, fasapi, djanao-3.0 asgi, aiohttp, etc...

The mainstream django framework in python is moving towards asynchronism. I have to say that it is going to enter the era of asynchronism.

The core focuses on asynchronous programming to improve performance.

Here are three main points to talk about:

• Synergetic process
• asyncio module for asynchronous programming
• Practical cases

Synergetic process

It is not provided by computer. Computer provides the concept of process and thread, and it is our programmer. Also known as micro thread, a technology of context switching in user state.

Through a thread to run between code swims and switches.

For example:
Ordinary execution

def func1():
    print(1)
    
    print(2)

def func(2):
    print(3)
    
    print(4)
    
func1()
func2()

result:

1
2
3
4

Implementation process

• greenlet, early modules
• yield keyword
• asynico decorator (py.34)
• asny, await keywords (py3.5) [official recommendation]

greenlet implementation

download

pip3 install greenlet

from greenlet import greenlet

def func1(): 
    print(1)            # Step 2 output 1
    gr2.switch()        # Step 3 switch to func2 function execution
    print(2)            # Step 6 output 2
    gr2.switch()        # Step 7 switch to func2 function execution

def func2():
    print(3)            # Step 4 output 3 
    gr1.switch()        # Step 5 switch to func1 function execution
    print(4)            # Step 8 output 4


gr1 = greenlet(func1)
gr2 = greenlet(func2)

gr1.switch() # Step 1: execute func1 function

result:

1
3
2
4

yield keyword

def func1():

    yield 1
    yield from func2()
    yield 2



def func2():
    yield 3
    yield 4


f1 = func1()
for item in f1:
    print(item)

result

1
3
2
4

You can understand the implementation of yeld

Play asyncio

Must be at or after Python 3.4 to use

import asyncio

@asyncio.coroutine
def func1():
    print(1)
    # Now I use sleep(2) for gain and loss. If I change it to network request, it will be more meaningful 
    yield from asyncio.sleep(2) # In case of IO time-consuming operation, automatically switch to other tasks in tasks
    print(2)

@asyncio.coroutine
def func2():
    print(3)
    yield from asyncio.sleep(2)
    print(4)


tasks = [
    asyncio.ensure_future(func1()),
    asyncio.ensure_future(func2())

]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))

# I should have waited for 4 seconds to execute, but I didn't wait for 4 seconds 

be careful ⚠️ : auto switch in case of io block
result

1
3
2
4

Async & await keyword

Available in Python 3.5 and later

import asyncio

async def func1():
    print(1)
    await asyncio.sleep(2)
    print(2)


async def func2():
    print(3)
    await asyncio.sleep(2)
    print(4)

tasks = [
    asyncio.ensure_future(func1()),
    asyncio.ensure_future(func2())

]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))

The most common keyword is async & await

The significance of synergism

If an io wait time is encountered in a thread, the thread will not wait all the time. Use the idle time to perform other tasks. If that task has a result, it will go back to get the result.

Case:
Network io requests three pictures

Normal mode (synchronous programming)

pip3 install requests

import requests

def download_image(url):
    print(f'Start downloading pictures:{url}')
    response = requests.get(url)
    print(f'Download complete')
    file_name = url.rsplit("/")[-1]
    with open(file_name,'wb',) as f:
        f.write(response.content)



if __name__ == '__main__':
    url_list = [
        'http://a3.att.hudong.com/14/75/01300000164186121366756803686.jpg',
        'http://a2.att.hudong.com/36/48/19300001357258133412489354717.jpg',
        'http://a0.att.hudong.com/64/76/20300001349415131407760417677.jpg',
    ]
    for url in url_list:
        download_image(url)

result:

Start downloading pictures: http://a3.att.hudong.com/14/75/01300000164186121366756803686.jpg
 Download complete
 Start downloading pictures: http://a2.att.hudong.com/36/48/19300001357258133412489354717.jpg
 Download complete
 Start downloading pictures: http://a0.att.hudong.com/64/76/20300001349415131407760417677.jpg
 Download complete

It turns out that it is executed in sequence. If your request is two minutes, you need to wait six minutes.
If we do it asynchronously, it will greatly improve the performance.

Co programming mode (asynchronous programming)

import asyncio
import aiohttp

async def fetch(session,url):
    print(f'Send request{url}')
    async with session.get(url,verify_ssl=False) as response:

        content =  await response.read()
        file_name = url.rsplit('/')[-1]
        with open(file_name,'wb') as f:
            f.write(content)
    print('Download complete')

async def main():
    async with aiohttp.ClientSession() as session:
        url_list = [
            'http://a3.att.hudong.com/14/75/01300000164186121366756803686.jpg',
            'http://a2.att.hudong.com/36/48/19300001357258133412489354717.jpg',
            'http://a0.att.hudong.com/64/76/20300001349415131407760417677.jpg',
        ]
        tasks = [
            asyncio.create_task(fetch(session,url)) for url in url_list
        ]
        await asyncio.wait(tasks)

if __name__ == '__main__':
    asyncio.run(main())

result:

Send request http://a3.att.hudong.com/14/75/01300000164186121366756803686.jpg
 Send request http://a2.att.hudong.com/36/48/19300001357258133412489354717.jpg
 Send request http://a0.att.hudong.com/64/76/20300001349415131407760417677.jpg
 Download complete
 Download complete
 Download complete