python's requests module can simulate browser requests. Compared with urllib, the api of requests module is more convenient (essentially encapsulating urllib3).

Installation mode

pip install requests

GET-based requests

response = requests.request("get", "http://www.baidu.com/")

If you want to add headers, you can pass in the headers parameter to increase the headers information in the request header. If you want to pass parameters in the url, you can use params parameters.

import requests

kw = {'wd': 'Polar bear'}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/54.0.2840.99 Safari/537.36"}

# params receives the query parameters of a dictionary or string, and the dictionary type is automatically converted to url encoding without the need of urlencode().
response = requests.get("http://www.baidu.com/s?", params=kw, headers=headers)

# Looking at the response content, response.text returns data in Unicode format
print(response.text)

# View the response content, byte stream data returned by response.content
print(response.content)

# View the full url address
print(response.url)

# View response header character encoding
print(response.encoding)

# View the response code
print(response.status_code)

• When using response.text, Requests automatically decodes response content based on HTTP response text encoding, and most Unicode character sets can be seamlessly decoded. The result you print may be scrambled. Just call the encoding method in front and set it to'utf-8'.
• When using response.content, the original binary byte stream of server response data is returned, which can be used to save binary files such as pictures.

POST-based requests

For post requests, we usually pass with parameters, or even with data.

import requests

formdata = {
    "type": "AUTO",
    "i": "i love python",
    "doctype": "json",
    "xmlVersion": "1.8",
    "keyfrom": "fanyi.web",
    "ue": "UTF-8",
    "action": "FY_BY_ENTER",
    "typoResult": "true"
}

url = "http://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule&smartresult=ugc&sessionFrom=null"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/51.0.2704.103 Safari/537.36"}

response = requests.post(url, data=formdata, headers=headers)

print(response.text)

# If it is a json file, it can be displayed directly
print(response.json())