Use of python crawler selenium

Keywords: Python Selenium crawler

brief introduction

- Selenium is one of the most widely used open source Web UI (user interface) automation test suites.
----Let my program connect to the browser and let the browser complete various complex operations. We only accept the final result
-----selenium; Automated test tools
----You can open the browser. Then operate the browser like a person. Programmers can directly extract all kinds of information on Web pages from selenium

----Environment construction:
pip install selenium -i Tsinghua source

----Download browser driver; https://npm.taobao.org/mirrors/chromedriver
----Put the extracted browser driver chromedriver in the folder where the python interpreter is located
----Let selenium launch Google browser

Use of selenium webdriver

----First, create a browser object (take Chrome as an example)
driver = Chrome()

  1. Get web pages. There are two ways to get web pages:

----Use the Get method - driver.get("www.yiibai.com");

----Use the navigate method - driver. Navigate(). To(“ https://yiibai.com/selenium/ ");

  1. Find the form and send user input
web.find_element_by_xpath('//[@id="JuserName"]').send_keys('***')
  1. Clear user input
    The clear() method is used to clear user input from the text box.
driver.find_element_by_xpath().clear();
  1. Get data through Web elements
    Sometimes you need to get text written through a web element to perform some assertions and debugging. Use the getText() method to get the data written through any web element.
driver.find_element_by_xpath(**).getText();
  1. Execute Click event
    The click() method is used to perform a click operation on any Web element.
driver.find_element_by_xpath(**).click();
  1. Navigate backward in browser history
driver.navigate().back();
  1. Navigate forward in browser history
driver.navigate().forward();
  1. Refresh / reload web pages
driver.navigate().refresh();
  1. Close browser
driver.close();
  1. Close the browser and all other windows associated with the driver
driver.quit();
  1. Move between Windows
driver.switch_to_window(web.window_handles[])

Return to previous window

web.switch_to_default_content()
  1. Move between frame s
web.switch_to_frame(**)
  1. Drag and drop
    Use the Action class to perform drag and drop operations.
WebElement element = driver.findElement(By.name("source"));  
WebElement target = driver.findElement(By.name("target"));  

14. Implementation of headless browser
opt = Options()

# opt.add_argument("--headless")
# opt.add_argument("--disable-gpu")
opt.add_argument('--disable-blink-features=AutomationControlled')

#1. Create browser object
web = Chrome(options=opt)

Implement 12306 drag login authentication instance


from selenium.webdriver.chrome.options import Options
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
import time

from selenium.webdriver.common.action_chains import ActionChains
opt = Options()
# opt.add_argument("--headless")
# opt.add_argument("--disable-gpu")
opt.add_argument('--disable-blink-features=AutomationControlled')

#1. Create browser object
web = Chrome(options=opt)
# 2. Open a website
web.get("https://kyfw.12306.cn/otn/resources/login.html")
print(web.title)
web.find_element_by_xpath('//*[@id="toolbar_Div"]/div[2]/div[2]/ul/li[2]/a').click()
web.find_element_by_xpath('//*[@id="J-userName"]').send_keys('user name ')
web.find_element_by_xpath('//*[@id="J-password"]').send_keys('password ')
web.find_element_by_xpath('//*[@id="J-login"]').click()
time.sleep(3)
index = web.find_element_by_xpath('//*[@id="nc_1__scale_text"]/span')
ActionChains(web).drag_and_drop_by_offset(index, 300, 0).perform()


Using super eagle to log in to super Eagle website image verification example

from chaojiying import Chaojiying_Client

from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
web = Chrome()
web.get("https://www.chaojiying.com/user/login/")

image = web.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/form/div/img").screenshot_as_png

chaojiying = Chaojiying_Client('user name', 'password', 'id')
print(chaojiying.PostPic(image, 1902))
dic = chaojiying.PostPic(image, 1902)
vs = dic['pic_str']

web.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/form/p[1]/input").send_keys('user name')
web.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/form/p[2]/input").send_keys('password')
web.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/form/p[3]/input").send_keys(vs,Keys.ENTER)

Posted by Jragon on Fri, 01 Oct 2021 13:02:13 -0700