python crawler 01 - create a simple crawler (with 100G of a novel web database)

Disclaimer: the database provided in this article is for technical verification only, and any form of commercial and reprint activities based on this database are prohibited. The legal liability arising therefrom shall be borne by itself! , once you continue to read this article, you will be deemed to agree to this stat ...

Posted by noobie_daddy on Wed, 01 Apr 2020 18:59:41 -0700

Python+Selenium notes (12): data driven testing

(1) Foreword Through the use of data-driven testing, the input value and expected results can be parameterized. (for example: input data and expected results can directly read the data of Excel document) (2) ddt The ddt library can parameterize the variables in the test. When using ddt, use @ ddt decorator on the test class and @ data decorator ...

Posted by viko20 on Tue, 31 Mar 2020 17:41:15 -0700

Python 3 reptile (15) agent

 Infi-chu: http://www.cnblogs.com/Infi-chu/ 1, Set up proxy 1.urllib #HTTP proxy type from urllib.error import URLError from urllib.requests import ProxyHandler,build_opener proxy='127.0.0.1:9743' # proxy='username:password@127.0.0.1:9743 'username and password are placed at the beginning proxy_handler=ProxyHandler({ 'http':'http://'+proxy, ...

Posted by minus4 on Fri, 20 Mar 2020 08:45:12 -0700

Crawler requests Library

If you want to use selenium to realize the functions of B station automatic login and click like, you can check how to solve the sliding unlocking. However, it's about the content of the crawler, and then you start to learn about the crawler. Before long, you want to make the website that records your life, so your friends recommend the layui f ...

Posted by madwormer2 on Sat, 14 Mar 2020 08:23:12 -0700

Crawler requests Library

If you want to use selenium to realize the functions of B station automatic login and click like, you can check how to solve the sliding unlocking. However, it's about the content of the crawler, and then you start to learn about the crawler. Before long, you want to make the website that records your life, so your friends recommend the layui f ...

Posted by JMJimmy on Sat, 14 Mar 2020 08:22:52 -0700

Then climb the hook, and directly ignore the anti climbing! Selenium+Xpath+re visible and creepable

Then climb the hook, and directly ignore the anti climbing! Selenium+Xpath+re visible and creepable I wrote a blog before python successfully climbs the dragnet -- initial understanding of anti climbing (a small white real climbing path, the content is a little bit smaller) This is the first time for ...

Posted by Imad on Mon, 09 Mar 2020 01:23:42 -0700

How to crawl all the cat's eye information (movie information, actor information)

Crawl all the information of Cat's Eye, mainly refer to the movie information and actors information in the movie list, such as the interface below. There are two difficulties when crawling.One: Font encryption (now seems to have a newer mechanism, but it is not possible to use the online method); ...

Posted by zevious on Tue, 03 Mar 2020 18:39:12 -0800

Splitting and Extracting Text Data in pandas

This paper mainly shares the splitting, extracting and merging of text data to prepare for the next visual analysis. The data comes from the employment information of boss and dragnet data analysis positions, totaling 9458. The crawling methods of the pull-hook are as follows: Python selenium+beautifuls ...

Posted by timelf123 on Mon, 24 Feb 2020 19:58:24 -0800

Shocked, Python cracked the BiliBili slider verification code to avoid human-computer identification perfectly

Python cracking BiliBili slider verification code |Perfect is impossible, add a shock! Python breaks the BiliBili slider verification code, avoiding human-computer identification perfectly. It can have Preparation Landing page of station B https://passport.bilibili.com/login python3 pip install sel ...

Posted by gunslinger008 on Sun, 16 Feb 2020 04:38:11 -0800

[Python Crawler] 8. Selenium and PhantomJS for Dynamic HTML Processing

Summary of Expiration: [Python Crawler] 1. Requests and Responses of HTTP and HTTPS of Crawler Principles [Python Crawler] 2. Definition, classification, flow and encoding format of crawler principles [Python Crawler] 3. Requests HTTP Library for Data Grabbing [Python Crawler] 4. Fiddler, ...

Posted by digitalecartoons on Sat, 15 Feb 2020 18:03:25 -0800