Implantation agent

1--file crawlIPpoll
2--Implantation agent middlewares.py file
3--setting open
DOWNLOADER_MIDDLEWARES = {
   'XXXPro.middlewares.ProxyDownloaderMiddleware': 543,
   'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

-- multi task
 Multiple tasks running on the same operating system at the same time

Serial: multiple execution in sequence
 Parallel: multiple simultaneous execution requires multiple machines to support cpu
 Concurrent: multiple tasks are executed alternately on the same cpu



--Multi task technology implementation

----Multiprocess
 Explanatory language is C
 Object oriented for application layer development

When the program is executed, the operating system will create a process, which is the unit of program execution
 A process has a process id, machine source code and various resources;
When the process is executed, the binary machine source code of the program is first put into the memory code segment
 Then the system assigns the process to the program and adds the process id, which is assigned according to the instructions of the machine source code
 Memory resources and I/O, and then let the process and other processes occupy the CPU (processor) in the form of concurrency
 , once it is captured, it will be executed

Process is the basic unit of operating system resource allocation

----Multithreading
 When the program is redesigned, multithreading is generally used
 Because process creation requires constant allocation of memory and I/O, which is wasteful
 Thread is the basic execution unit of a process
 A process contains a main thread and several sub threads
 After the operating system has created a process and allocated resources, it will create a main thread,
And let the main thread participate in the scheduling of the processor
 For some time-consuming operations, to prevent these tasks from blocking the main thread,
Often, a sub thread will be opened to perform these tasks. The sub process and the main process are concurrent,
So as to improve the performance of the system

Three ways to create multithreading

1--Guardian creation--Main end sub end
def thread1():
    _thread.start_new_thread(run,("MrWang",))
    name = threading.current_thread().name
    print("Main thread:",name)
    for i in range(1,9):
        print("Main thread%s Implemented%d Secondary cycle"%(name,i+1))
        sleep(1)
        
2--Non guard creation--Main endchild continue
def thread2():
    t = threading.Thread(target=run,args=("zz",))
    t.start()
    name = threading.current_thread().name
    print("Main thread:", name)
    for i in range(1, 9):
        print("Main thread%s Implemented%d Secondary cycle" % (name, i + 1))
        sleep(1)
        
3--Custom creation--Main endchild continue
class MyThead(threading.Thread):
    # Override construction method
    def __init__(self,name,task):
        super().__init__()
        self.name = name
        self.task = task
    # Override run method
    def run(self):
        print("The current sub thread is:",self.name,self.task)
        for i in range(1, 9):
            print("Current child thread%s Implemented%d Secondary cycle" % (self.name, i + 1))
            sleep(2)
def thread3():
    t = MyThead("A","B")
    t.start()
    name = threading.current_thread().name
    print("Main thread:", name)
    for i in range(1, 9):
        print("Main thread%s Implemented%d Secondary cycle" % (name, i + 1))
        sleep(1)

---Synchronization of threads
# Synchronization is serial. All tasks are added to the same synchronization queue. The previous tasks cannot be opened until they are completed
t1.join()

---Thread conflicts
1,Join synchronization queue with thread synchronization
t.join()#serial#Cause thread blocking

2,Lock resources(mutex)
lock = threading.Lock()
def increase2():
    if lock.acquire():
        for i in range(1000000):
            m += 1
        print(m)
    lock.release()
# Synchronization lock condition = threading.Condition() can't be mutually exclusive when reading and writing

---Semaphore
//Limit the maximum concurrent amount 
sem = threading.Semaphore(3)

Queue first in first out

#Definition q = Queue(3)