This lock is also called a recursive lock.

3. Semaphore (semaphore)
This is also a lock, you can specify how many threads can get the lock at the same time, up to five (the mutex mentioned above can only be obtained by one thread)

import threading
import time

semaphore=threading.Semaphore(5)

def foo():
    semaphore.acquire()
    time.sleep(2)
    print('ok')
    semaphore.release()

for i in range(10):
    t=threading.Thread(target=foo,args=())
    t.start()

The result is to print five ok every two seconds.

4.Event objects
Threads run independently. Event objects are needed if there is communication between threads, or if a thread needs to perform the next operation according to the state of a thread. Event object can be regarded as a flag bit, the default value is false. If a thread waits for an Event object and the flag bit in the Event object is false at this time, the thread will wait until the flag bit is true. After the flag bit is true, all threads waiting for the Event object will be awakened.

event.isSet(): Returns the status value of event;

event.wait(): If event.isSet()==False, the thread will be blocked.

event.set(): Set the state value of event to True, and all blocking pool threads are activated into ready state, waiting for the operating system to schedule; when setting objects, the default is False

event.clear(): Restore event's status value to False.

An example is given to illustrate the use of Event objects:

import threading,time

event=threading.Event()     #Create a event object

def foo():
     print('wait.......')
     event.wait()
     #event.wait(1)#If the flag bit in the event object is Flase, then blocking
     #wait()The parameter inside means: just wait for 1 second, if the flag has not been changed after 1 second, then wait, continue to execute the following code
     print('connect to redis server')

print('attempt to start redis sever)')

time.sleep(3)
event.set()

for i in range(5):
     t=threading.Thread(target=foo,args=())
     t.start()
#3 Seconds later, the main thread ends, but the sub-thread is not the daemon thread, the sub-thread is not finished, so the program does not end, should be 3 seconds later, set the flag to true，Namely event.set()

5. Queues

Official documents say queues are very useful for data security in multithreading

Queues can be understood as a data structure that stores and reads and writes data. It's like adding a lock to the list.

5.1 get and put methods

import queue
#Only read and write data in the queue put and get Two methods, none of the list methods.
q=queue.Queue()#Create a queue object  FIFO FIFO
#q=queue.Queue(20)
#There can be a parameter that sets the maximum amount of data stored, which can be understood as the maximum number of grids.
#If the parameter is set to 20, at the 21st put, the program will block until there is an empty location, that is, data is moved by get.

q.put(11)#Put value
q.put('hello')
q.put(3.14)

print(q.get())#Value 11
print(q.get())#Value hello
print(q.get())#Value 3.14
print(q.get())#Block, wait put A Data

There is a default parameter block=True in the get method. Change this parameter to False, and queue.Empty will be wrong if the value is not reached.

This is equivalent to q.get_nowait())

5.2 join and task_done methods

Join is used to block processes, and it makes sense to use it in conjunction with task_done. Event object can be used to understand, no times put(), join counter plus 1, no times task_done (), counter minus 1, counter is 0, before the next put()

Notice that task_done is added after each get().

import queue
import threading
#Only in the queue put and get Two methods, none of the list methods.
q=queue.Queue()#
def foo():#Storage of data
    # while True:
    q.put(111)
    q.put(222)
    q.put(333)
    q.join()
    print('ok')#There is join，This is where the program stops.
def bar():
    print(q.get())
    q.task_done()
    print(q.get())
    q.task_done()
    print(q.get())
    q.task_done()#In each get()After the statement, add
t1=threading.Thread(target=foo,args=())
t1.start()
t2=threading.Thread(target=bar,args=())
t2.start()

#t1,t2 It doesn't matter who comes first or who comes next, because it will block and wait for the signal.

5.3 Other methods

The queues mentioned above are FIFO mode, and there are LIFO mode and priority queue.

Classqueue. LifoQueue (maxsize)

Priority queues are written as: class queue.Priorityueue(maxsize)

IV. Multiprocess

There's a global lock (GIL) in python that prevents multithreading from using multicores, but if it's a multiprocess, the lock won't be limited. How to open multiple processes, you need to import a multiprocessing module

import multiprocessing

import time

def foo():
    print('ok')
    time.sleep(2)

if __name__ == '__main__':#Must be in this format
    p=multiprocessing.Process(target=foo,args=())
    p.start()
    print('ending')

Although multi-process can be opened, we must pay attention to not too much, because inter-process switching consumes system resources very much. If thousands of sub-processes are opened, the system will collapse, and inter-process communication is also a problem. Therefore, processes can be used or not, and processes can be used less or less.

1. Interprocess communication

There are two ways to communicate between processes, queue and pipeline.

1.1 Interprocess queues

Each process is a separate piece of space in memory. It can share data without threads, so the queue can only be passed from parent process to child process by parameter.

import multiprocessing
import threading

def foo(q):
    q.put([12,'hello',True])

if __name__ =='__main__':
    q=multiprocessing.Queue()#Create process queues

    #Create a subthread
    p=multiprocessing.Process(target=foo,args=(q,))
    #Passing this queue object to the parent process by reference
    p.start()

    print(q.get())

1.2 Pipeline

The socket that I learned before is actually a pipe. The sock of the client and conn of the server are the two ends of the pipe. This is also the way to play in the process. There should also be two ends of the pipe.

from multiprocessing import  Pipe,Process

def foo(sk):
    sk.send('hello')#The main process sends messages
    print(sk.recv())#The main process receives messages

sock,conn=Pipe()#Two ends of the pipe were created.
if __name__ == '__main__':

    p=Process(target=foo,args=(sock,))
    p.start()

    print(conn.recv())#Subprocesses receive messages
    conn.send('hi son')#Subprocesses send messages

2. Data Sharing between Processes

We have realized communication between processes through process queue and pipeline, but we have not realized data sharing yet.

Data sharing between processes needs to be implemented by referencing a manager object. All data types used are created by using manager points.

from multiprocessing import Process
from multiprocessing import Manager
def foo(l,i):
    l.append(i*i)

if __name__ == '__main__':
    manager = Manager()

    Mlist = manager.list([11,22,33])#Create a shared list

    l=[]
    for i in range(5):
        #Open up five sub-processes
        p = Process(target=foo, args=(Mlist,i))
        p.start()
        l.append(p)
    for i in l:
        i.join()#join The method is to wait for the process to finish before executing the next one.
    print(Mlist)

3. Process pool

The function of the process pool is to maintain a maximum amount of processes. If the maximum amount is exceeded, the program will block until the available processes are known.

from multiprocessing import Pool

import time

def foo(n):
    print(n)
    time.sleep(2)

if __name__ == '__main__':
    pool_obj=Pool(5)#Create a process pool

    #Creating processes through process pools
    for i in range(5):
        p=pool_obj.apply_async(func=foo,args=(i,))
        #p Is the pool object created
    # pool Use first close()，stay join(),Just remember.
    pool_obj.close()
    pool_obj.join()

    print('ending')

There are several methods in the process pool:

1.apply: Take a process from the process pool and execute it
 2.apply_async: asynchronous version of apply
 3.terminate: Close the thread pool immediately
 4.join: The main process waits for all subprocesses to finish executing, and must be after close or terminate
 5.close: Wait for all processes to finish before closing the thread pool

V. COOPERATION

In hand, the world I have, say go away. Knowing the coroutines, the process threads mentioned above will be forgotten.

There is no upper limit and the consumption between handoffs can be neglected.

1.yield

Think back and forth about the word "yield". Familiar with it, no, that's the one used by the generator. Yield is an amazing thing, which is a feature of Python.

Normally, the function stops when it encounters return, and then returns the value after return. By default, None, yield and return are very similar, but when it encounters yield, it does not stop immediately, but pauses until it encounters next(), (the principle of for loop is also next()). You can also follow a variable in front of the field by sending () to the field to store the value in the variable in front of the field.

import time

def consumer():#There is yield, which is a generator.
    r=""
    while True:
        n=yield r#Program pauses, waiting for next() signal
        # if not n:
        #     return

        print('consumer <--%s..'%n)
        time.sleep(1)
        r='200 ok'

def producer(c):
    next(c)#Activation generator c
    n=0
    while n<5:
        n=n+1
        print('produer-->%s..'%n)
        cr = c.send(n)#Send data to generator
        print('consumer return :',cr)
　　c.close() #When the production process is over, turn off the generator
if __name__ == '__main__':
    c=consumer()
    producer(c)

Look at the above example, the whole process does not appear lock, but also to ensure data security, more importantly, can control the order, elegant realization of concurrency, throw off multi-threaded streets.

Threads are called micro-processes, and co-processes are called micro-threads. The coroutine has its own register context and stack, so it can retain the state of the last call.

2.greenlet module

This module encapsulates yield, which makes program switching very convenient, but it can not achieve the function of value transfer.

from greenlet import greenlet

def foo():
    print('ok1')
    gr2.switch()
    print('ok3')
    gr2.switch()
def bar():
    print('ok2')
    gr1.switch()
    print('ok4')

gr1=greenlet(foo)
gr2=greenlet(bar)

gr1.switch()#start-up

3.gevent module

On the basis of greenlet module, a better module gevent is developed.

gevent provides better collaboration support for Python. Its basic principles are:

When a Greenlet encounters an IO operation, it will automatically switch to another greenlet, wait for the IO operation to complete, and then switch back, so as to ensure that there is always Greenlet running, not waiting.

import requests
import gevent
import time
def foo(url):

    response=requests.get(url)
    response_str=response.text

    print('get data %s'%len(response_str))

s=time.time()
gevent.joinall([gevent.spawn(foo,"https://itk.org/"),
                gevent.spawn(foo, "https://www.github.com/"),
                gevent.spawn(foo, "https://zhihu.com/"),])

# foo("https://itk.org/")
# foo("https://www.github.com/")
# foo("https://zhihu.com/")
print(time.time()-s)

4. Advantages and disadvantages of the protocol:

Advantage:

Context switching consumes less

Convenient switching of control flow and simplified programming model

High concurrency, high scalability and low cost

Disadvantages:

Multicore is unavailable

Blocking operations block the entire program

VI. IO Model

Let's compare four IO models

1.blocking IO

2.nonblocking IO

3.IO multiplexing

4.asynchronous IO

We take IO for example. It involves two system objects, one is the thread or process that calls the IO, the other is the system kernel. When reading data, it will go through two stages:

Waiting for data preparation

The process of copying data from the kernel state to the user state (because the data transmission of the network is realized by the physical device, which is hardware and can only be processed by the kernel state of the operating system, but the reading data is used by the program, so the switch of this step is needed).

1.blocking IO (Blocking IO)

The typical read operation is shown below.

Under linux, the default sockets are blocking. Looking back on the sockets we used before, sock and conn are two connections. The server can only monitor one connection at the same time, so if the server is waiting for the client to send messages, other connections can not connect to the server.

In this mode, waiting for data and replicating data all need to wait, so the whole process is blocked.

2.nonlocking IO (non-blocking IO)

After the server establishes the connection, with this command, it becomes a non-blocking IO mode.

In this mode, if there is data, it will fetch it, and if there is no error, it can add an exception capture. It is not blocked while waiting for data, but it is blocked when copy ing data.

The advantage is that the waiting time can be used, but the disadvantage is also obvious: there are many system calls, which consume a lot; and when the program does something else, the data arrives, although it will not be lost, but the data received by the program is not real-time.

3.IO multiplexing (IO multiplexing)

This is commonly used. The accept() we used before has two functions:

1. Monitor, Wait for Connection

2. Connecting

Now let's replace the first role of accept with select. The advantage of select is that it can listen to many objects, no matter which object activity is, it can react and collect the active objects into a list.

import socket
import select
sock=socket.socket()
sock.bind(('127.0.0.1',8080))
sock.listen(5)


inp=[sock,]
while True:
    r=select.select(inp,[],[])
    print('r',r[0])
    for obj in r[0]:

        if obj == sock:
            conn,addr=obj.accept()

But the function of establishing connection is accept. With this, we can realize tcp chat in a concurrent way.

 1 # Server
 2 import socket
 3 import time
 4 import select
 5 
 6 sock=socket.socket()
 7 sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
 8 sock.bind(('127.0.0.1',8080))
 9 sock.listen(5)
10 
11 inp=[sock,]#List of listening socket objects
12 
13 while True:
14     r=select.select(inp,[],[])
15     print('r',r[0])
16     for obj in r[0]:
17         if obj == sock:
18             conn,addr=obj.accept()
19             inp.append(conn)
20         else:
21             data=obj.recv(1024)
22             print(data.decode('utf8'))
23             response=input('>>>>:')
24             obj.send(response.encode('utf8'))

Only when a connection is established, the sock is active, and the list will have this object. If after the connection is established, the active object is not sock, but conn in the process of sending and receiving messages. So in practice, it is necessary to judge whether the object in the list is sock.

In this model, the process of waiting for data and copy ing data is blocked, so it is also called full blocking. Compared with blocking IO model, the advantage of this model is to handle multiple connections.

In addition to select, IO multiplexing has two ways, poll and epoll.

Only select is supported under windows, but in linux, all three are supported. Epoll is the best, the only advantage of select is that it can be used on many platforms, but the disadvantage is also obvious, that is, the efficiency is very poor. poll is the intermediate transition between epoll and select. Compared with select, there is no limit to the number of polls that can be monitored. There is no maximum connection limit for epoll, and the monitoring mechanism is completely changed. The selection mechanism is polling (every data is checked once, even if it finds a change, it will continue to check). The epoll mechanism is a callback function, which calls the callback function if any object changes.

4. Asynchronous IO (Asynchronous IO)

This mode is non-blocking in the whole process, only non-blocking in the whole process can be called asynchronism. Although this mode looks good, but in practice, if the request volume is large, the efficiency will be very low, and the task of the operating system is very heavy.

Seventh, selectors module

If you learn this module, you don't need to use select, poll, or epoll. Their interfaces are all this module. We just need to know how to use this interface and what it encapsulates, without considering it.

In this module, the binding of sockets and functions uses a regesier() method. The usage of the module is very fixed. The example of the server side is as follows:

 1 import selectors,socket
 2 
 3 sel=selectors.DefaultSelector()
 4 
 5 sock=socket.socket()
 6 sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
 7 sock.bind(('127.0.0.1',8080))
 8 sock.listen(5)
 9 sock.setblocking(False)
10 
11 def read(conn,mask):
12     data=conn.recv(1024)
13     print(data.decode('utf8'))
14     res=input('>>>>>>:')
15     conn.send(res.encode('utf8'))
16 
17 def accept(sock,mask):
18     conn,addr=sock.accept()
19     sel.register(conn,selectors.EVENT_READ,read)#conn and read Function binding
20 #Binding socket objects and functions
21 #Binding( register)Meaning, socket objects conn When a change occurs, the bound function can be executed
22 sel.register(sock,selectors.EVENT_READ,accept)#The middle one is fixed.
23 while True:
24     events=sel.select() #Listen for socket objects (the one registered)
25     #The following lines of code are basically fixed.
26     # print('events',events)
27     for key,mask in events:
28         callback = key.data#Binding functions,
29         # key.fileobj Is the active socket object
30         # print('callback',callable)
31         #mask It's fixed.
32         callback(key.fileobj,mask)#callback Is a callback function
33         # print('key.fileobj',key.fileobj)

Next article, database