Python thread pool / process pool memory management

concurrent.futures thread pool / process pool memory management cause In the past, there was a need for a crawler task. A machine with 1G memory ran a crawler and crawled the data of a website. Previously, this method was used: futures = list() with ThreadPool(max_workers=20) as exc: for tr in table.selec ...

Posted by knowj on Fri, 03 Apr 2020 00:03:35 -0700

Usage of urlib Library of Python crawler (1): crawling, saving pages, obtaining request information

import urllib.request I. Introduction Urlib is Python's built-in HTTP request library, which includes the following modules: urllib.request: request module urllib.error: exception handling module urllib.parse: url parsing module Urllib.robot parser: robot.txt parsing module 2, Crawler specified URL with urllib.reque ...

Posted by bschwarz on Thu, 02 Apr 2020 22:36:14 -0700

Course assignment - get all campus news

Operation requirements: Take out all the news of a news list page and wrap it into a function. Get the total number of news articles and calculate the total number of pages. Get all news details for all news list pages. Find a topic that you are interested in, do data crawling and word segmentation analysis. Can't be the same as other students ...

Posted by zubinkasad on Thu, 02 Apr 2020 22:18:00 -0700

Python 2 to Python 3 code conversion tool 2to3.py

I've been meeting the need to switch from Python 2 code to Python 3 code before. I just recently learned that Python has an official conversion tool: 2to3.py.  Whether you download Python from the python official website or use Anaconda for installation, the directory is in {Python_HOME}\Tools\scripts. Run the 2to3 ...

Posted by kalaszabi on Thu, 02 Apr 2020 21:02:02 -0700

A python implementation of lock file

If multiple processes or independent programs want to write the same file, it is possible for everyone to write the file at the same time, which is not good, and the data may have problems. Recently, an open-source python implementation was found on the Internet, which is effective and concise. List and analyze the following co ...

Posted by corrupshun on Thu, 02 Apr 2020 20:16:57 -0700

python version control

Reference article: Thanks to Blogger 1 There are three levels of python version control: system level, folder level and pycharm level system level First, check the number of python versions installed in the system: Method 1: zht@zht0:~$ ls /usr/bin/python* /usr/bin/python /usr/bin/python2 /usr/bin/python2.7 /usr/b ...

Posted by pob123 on Thu, 02 Apr 2020 19:34:11 -0700

Python learning notes (9) Python tuples and dictionaries (2)

What is a dictionary A dictionary is another variable container model and can store any type of object. Colon is used for every key value = > value pair of the dictionary: divide, comma is used between every key value pair, divide, and the whole dictionary is included in curly bracket {} The key must be unique, but the value is not. Value ca ...

Posted by felipeebs on Thu, 02 Apr 2020 19:04:11 -0700

Day 3 of reading "Python programming from introduction to practice"

User:Hello I'm the forest Date:2018-03-16 Mark:Book " Python Programming from introduction to practice Python programming from introduction to practice Chapter IV (II) Use list Section To create a slice, specify the index of the first and last element to use. Like the function range(), Python stops after reaching the e ...

Posted by vivek on Thu, 02 Apr 2020 09:38:08 -0700

python upload file

Using python to bulk upload files Recently, I need to fill in the form and upload the file on the Internet, so I thought whether I could write a step to do it, so I finally completed this task through Baidu. Let's summarize Grab a bag You can see from the packet capture that the upload process is actually complete ...

Posted by lady_bug on Thu, 02 Apr 2020 08:13:37 -0700

Old boy homework day1 - user login

The first assignment I wrote after the old boy's training Demand: 1. Users can log in by user name and password; 2. Lock the user after three incorrect password entries 3. Put the user name, password and locked user in the file 4. If you don't have the user prompt, you haven't registered 5. Login with the locked user will prompt that the user h ...

Posted by shanx24 on Thu, 02 Apr 2020 05:01:47 -0700