If there is any error, please point out in the comment area. The author will change it immediately. The article is moved from my personal CSDN, so there is no watermark

Environmental description

• Operating system: Windows7 SP1 64
• python version: 3.7.7
• Browser: Google browser
• Browser version: 80.0.3987 (64 bit)
• Google browser driver: the driver version needs to correspond to the browser version. Different browsers use different drivers, Click download
• If you are using Firefox, check the version of Firefox, click Download address of GitHub Firefox driver Download (for students who are not good at English, right-click to translate, and each version has the instructions for using the corresponding browser version, just see the download clearly)

brief introduction

Selenium is an overall project that covers a series of tools and libraries that support the automation of Web browsers. And when automating, the operation will be the same as the real user operation.

Selenium has three versions: Selenium 1.0, selenium 2.0 and selenium 3.0;

Selenium 1.0 mainly calls JS to inject into the browser; at the beginning, Jason Huggins, the author of selenium, developed JavaScript test runner as a test tool, which was shown to many colleagues at that time (the author is also a very interesting soul). As you can see from the name of this test tool, it is a javascript based test. This tool is the "predecessor" of selenium.

Selenium 2.0 is based on the API provided by WebDriver to operate the elements of browser. WebDriver is a testing framework or an integrated API library.

Selenium 3.0 is extended based on Selenium 2.0, and there is little difference between them; this article will explain the technology with selenium 3.0.

"Through WebDriver, Selenium supports all mainstream browsers in the market, such as chrome (ium), Firefox, Internet Explorer, Opera and Safari," the official introduction said

Simple start

After installing the environment, simply use selenium to let the browser open CSDN official website.
When you configure the environment, you need to pay attention to: you must configure the driver to the system environment, or drop it to the root directory of your python.

First, webdriver is introduced:

from selenium.webdriver import Chrome

Of course, you can:

from selenium import webdriver

The introduction method varies from person to person, and then different methods are used to create different instances.

from selenium.webdriver import Chrome
driver = Chrome()

perhaps

from selenium import webdriver
driver = webdriver.Chrome()

The general python syntax will not be repeated below.
As mentioned before, the driver needs to be configured in the system environment, but the driver path cannot be added to the system environment due to other reasons. Here is a solution:

from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'F:\python\dr\chromedriver_win32\chromedriver.exe')

Use executable here_ Path specifies the driver address, which is the location where my driver is stored. Of course, this location can be made according to your own needs, and it is more flexible; in order to better illustrate, this article uses absolute path incoming.

Firefox:

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("http://www.csdn.net")

Google Browser:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.csdn.net")

Firefox is different from Google only in instantiation method, and other operation methods are consistent.

webdriver is introduced at the beginning of the code. After the browser object is instantiated in the code, get method is used to request the URL and open the required URL.

Analysis of realization

see webdriver.py from selenium import webdriver:

import warnings

from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
from .remote_connection import ChromeRemoteConnection
from .service import Service
from .options import Options


class WebDriver(RemoteWebDriver):
    """
    Controls the ChromeDriver and allows you to drive the browser.

    You will need to download the ChromeDriver executable from
    http://chromedriver.storage.googleapis.com/index.html
    """

    def __init__(self, executable_path="chromedriver", port=0,
                 options=None, service_args=None,
                 desired_capabilities=None, service_log_path=None,
                 chrome_options=None, keep_alive=True):
        """
        Creates a new instance of the chrome driver.

        Starts the service and then creates new instance of chrome driver.

        :Args:
         - executable_path - path to the executable. If the default is used it assumes the executable is in the $PATH
         - port - port you would like the service to run, if left as 0, a free port will be found.
         - options - this takes an instance of ChromeOptions
         - service_args - List of args to pass to the driver service
         - desired_capabilities - Dictionary object with non-browser specific
           capabilities only, such as "proxy" or "loggingPref".
         - service_log_path - Where to log information from the driver.
         - chrome_options - Deprecated argument for options
         - keep_alive - Whether to configure ChromeRemoteConnection to use HTTP keep-alive.
        """
        if chrome_options:
            warnings.warn('use options instead of chrome_options',
                          DeprecationWarning, stacklevel=2)
            options = chrome_options

        if options is None:
            # desired_capabilities stays as passed in
            if desired_capabilities is None:
                desired_capabilities = self.create_options().to_capabilities()
        else:
            if desired_capabilities is None:
                desired_capabilities = options.to_capabilities()
            else:
                desired_capabilities.update(options.to_capabilities())

        self.service = Service(
            executable_path,
            port=port,
            service_args=service_args,
            log_path=service_log_path)
        self.service.start()

        try:
            RemoteWebDriver.__init__(
                self,
                command_executor=ChromeRemoteConnection(
                    remote_server_addr=self.service.service_url,
                    keep_alive=keep_alive),
                desired_capabilities=desired_capabilities)
        except Exception:
            self.quit()
            raise
        self._is_remote = False

    def launch_app(self, id):
        """Launches Chrome app specified by id."""
        return self.execute("launchApp", {'id': id})

    def get_network_conditions(self):
        return self.execute("getNetworkConditions")['value']

    def set_network_conditions(self, **network_conditions):
        self.execute("setNetworkConditions", {
            'network_conditions': network_conditions
        })

    def execute_cdp_cmd(self, cmd, cmd_args):
        return self.execute("executeCdpCommand", {'cmd': cmd, 'params': cmd_args})['value']

    def quit(self):
        try:
            RemoteWebDriver.quit(self)
        except Exception:
            # We don't care about the message because something probably has gone wrong
            pass
        finally:
            self.service.stop()

    def create_options(self):
        return Options()

The comments indicate that this is "creating a new instance of the chrome driver and creating an instance of the chrome driver.".

Only the parameters used in this article are listed here:

• executable_path: the PATH to the executable. If you use the default, the executable is assumed to be in the PATH, where PATH is the system environment root

In the process of selenium automation, the necessary step is to start the service. In the init initialization method, the following code is found:

self.service = Service(
            executable_path,
            port=port,
            service_args=service_args,
            log_path=service_log_path)
self.service.start()

The above code instantiates the Service class, passes in the relevant parameters, and then starts the Service. The main parameter here is executable_path, that is to start the driver. View Service class( selenium.service ):

from selenium.webdriver.common import service


class Service(service.Service):
    """
    Object that manages the starting and stopping of the ChromeDriver
    """

    def __init__(self, executable_path, port=0, service_args=None,
                 log_path=None, env=None):
        """
        Creates a new instance of the Service

        :Args:
         - executable_path : Path to the ChromeDriver
         - port : Port the service is running on
         - service_args : List of args to pass to the chromedriver service
         - log_path : Path for the chromedriver service to log to"""

        self.service_args = service_args or []
        if log_path:
            self.service_args.append('--log-path=%s' % log_path)

        service.Service.__init__(self, executable_path, port=port, env=env,
                                 start_error_message="Please see https://sites.google.com/a/chromium.org/chromedriver/home")

    def command_line_args(self):
        return ["--port=%d" % self.port] + self.service_args

Check the implementation of the start method of the base class (because the base class is too long, not all of them are displayed, the base class is in the selenium.webdriver.common In import service):

def start(self):
        """
        Starts the Service.

        :Exceptions:
         - WebDriverException : Raised either when it can't start the service
           or when it can't connect to the service
        """
        try:
            cmd = [self.path]
            cmd.extend(self.command_line_args())
            self.process = subprocess.Popen(cmd, env=self.env,
                                            close_fds=platform.system() != 'Windows',
                                            stdout=self.log_file,
                                            stderr=self.log_file,
                                            stdin=PIPE)
        except TypeError:
            raise
        except OSError as err:
            if err.errno == errno.ENOENT:
                raise WebDriverException(
                    "'%s' executable needs to be in PATH. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            elif err.errno == errno.EACCES:
                raise WebDriverException(
                    "'%s' executable may have wrong permissions. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            else:
                raise
        except Exception as e:
            raise WebDriverException(
                "The executable %s needs to be available in the path. %s\n%s" %
                (os.path.basename(self.path), self.start_error_message, str(e)))
        count = 0
        while True:
            self.assert_process_still_running()
            if self.is_connectable():
                break
            count += 1
            time.sleep(1)
            if count == 30:
                raise WebDriverException("Can not connect to the Service %s" % self.path)

It is found that:

try:
      cmd = [self.path]
      cmd.extend(self.command_line_args())
      self.process = subprocess.Popen(cmd, env=self.env,
                                      close_fds=platform.system() != 'Windows',
                                      stdout=self.log_file,
                                      stderr=self.log_file,
                                      stdin=PIPE)
except TypeError:
            raise
        except OSError as err:
            if err.errno == errno.ENOENT:
                raise WebDriverException(
                    "'%s' executable needs to be in PATH. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            elif err.errno == errno.EACCES:
                raise WebDriverException(
                    "'%s' executable may have wrong permissions. %s" % (
                        os.path.basename(self.path), self.start_error_message)
                )
            else:
                raise
        except Exception as e:
            raise WebDriverException(
                "The executable %s needs to be available in the path. %s\n%s" %
                (os.path.basename(self.path), self.start_error_message, str(e)))
        count = 0
        while True:
            self.assert_process_still_running()
            if self.is_connectable():
                break
            count += 1
            time.sleep(1)
            if count == 30:
                raise WebDriverException("Can not connect to the Service %s" % self.path)

Start the subprocess to start the driver. Receives and throws an exception and reports an error when an exception occurs. Turn on the driver to open the browser.

We know how selenium starts the service after exception throw detection. Next, see the implementation process of get request URL.
View webdriver base class( selenium.webdriver.remote.webdriver), find the get method:

def get(self, url):
    """
    Loads a web page in the current browser session.
    """
    self.execute(Command.GET, {'url': url})

def execute(self, driver_command, params=None):
        """
        Sends a command to be executed by a command.CommandExecutor.

        :Args:
         - driver_command: The name of the command to execute as a string.
         - params: A dictionary of named parameters to send with the command.

        :Returns:
          The command's JSON response loaded into a dictionary object.
        """
        if self.session_id is not None:
            if not params:
                params = {'sessionId': self.session_id}
            elif 'sessionId' not in params:
                params['sessionId'] = self.session_id

        params = self._wrap_value(params)
        response = self.command_executor.execute(driver_command, params)
        if response:
            self.error_handler.check_response(response)
            response['value'] = self._unwrap_value(
                response.get('value', None))
            return response
        # If the server doesn't send a response, assume the command was
        # a success
        return {'success': 0, 'value': None, 'sessionId': self.session_id}

Through the get method, we know that the execute method is called and the Command.GET And url.
see Command.GET Class command of( selenium.webdriver.remote.command) that command is a constant of the standard WebDriver command; find the get constant:

GET = "get"

From the file, it should be the class file that executes the command mode.
First, sort out the following process:

• Start service → call get method

The specific process of get method is as follows:

• The get method calls the execute method. The passed in parameter is Command.GET With url, check that the value of the command is a standard constant. In the execute method,

The implementation of execute is as follows:

def execute(self, driver_command, params=None):
        """
        Sends a command to be executed by a command.CommandExecutor.

        :Args:
         - driver_command: The name of the command to execute as a string.
         - params: A dictionary of named parameters to send with the command.

        :Returns:
          The command's JSON response loaded into a dictionary object.
        """
        if self.session_id is not None:
            if not params:
                params = {'sessionId': self.session_id}
            elif 'sessionId' not in params:
                params['sessionId'] = self.session_id

        params = self._wrap_value(params)
        response = self.command_executor.execute(driver_command, params)
        if response:
            self.error_handler.check_response(response)
            response['value'] = self._unwrap_value(
                response.get('value', None))
            return response
        # If the server doesn't send a response, assume the command was
        # a success
        return {'success': 0, 'value': None, 'sessionId': self.session_id}

The core code is:

params = self._wrap_value(params)
response = self.command_executor.execute(driver_command, params)
if response:
    self.error_handler.check_response(response)
    response['value'] = self._unwrap_value(
        response.get('value', None))
    return response

Main view:

self.command_executor.execute(driver_command, params)

Where command_executor is the instance after initialization. View the derived class webdriver (selenium import webdriver) command_ The instantiation of the executor is:

RemoteWebDriver.__init__(
                self,
                command_executor=ChromeRemoteConnection(
                    remote_server_addr=self.service.service_url,
                    keep_alive=keep_alive),
                desired_capabilities=desired_capabilities)

View the chromemoteconnection class (selenium import remote_connection):

from selenium.webdriver.remote.remote_connection import RemoteConnection


class ChromeRemoteConnection(RemoteConnection):

    def __init__(self, remote_server_addr, keep_alive=True):
        RemoteConnection.__init__(self, remote_server_addr, keep_alive)
        self._commands["launchApp"] = ('POST', '/session/$sessionId/chromium/launch_app')
        self._commands["setNetworkConditions"] = ('POST', '/session/$sessionId/chromium/network_conditions')
        self._commands["getNetworkConditions"] = ('GET', '/session/$sessionId/chromium/network_conditions')
        self._commands['executeCdpCommand'] = ('POST', '/session/$sessionId/goog/cdp/execute')

It is known that the base class initialization method is called, and the implementation of execute method is as follows:

def execute(self, command, params):
        """
        Send a command to the remote server.

        Any path subtitutions required for the URL mapped to the command should be
        included in the command parameters.

        :Args:
         - command - A string specifying the command to execute.
         - params - A dictionary of named parameters to send with the command as
           its JSON payload.
        """
        command_info = self._commands[command]
        assert command_info is not None, 'Unrecognised command %s' % command
        path = string.Template(command_info[1]).substitute(params)
        if hasattr(self, 'w3c') and self.w3c and isinstance(params, dict) and 'sessionId' in params:
            del params['sessionId']
        data = utils.dump_json(params)
        url = '%s%s' % (self._url, path)
        return self._request(command_info[0], url, body=data)

    def _request(self, method, url, body=None):
        """
        Send an HTTP request to the remote server.

        :Args:
         - method - A string for the HTTP method to send the request with.
         - url - A string for the URL to send the request to.
         - body - A string for request body. Ignored unless method is POST or PUT.

        :Returns:
          A dictionary with the server's parsed JSON response.
        """
        LOGGER.debug('%s %s %s' % (method, url, body))

        parsed_url = parse.urlparse(url)
        headers = self.get_remote_connection_headers(parsed_url, self.keep_alive)
        resp = None
        if body and method != 'POST' and method != 'PUT':
            body = None

        if self.keep_alive:
            resp = self._conn.request(method, url, body=body, headers=headers)

            statuscode = resp.status
        else:
            http = urllib3.PoolManager(timeout=self._timeout)
            resp = http.request(method, url, body=body, headers=headers)

            statuscode = resp.status
            if not hasattr(resp, 'getheader'):
                if hasattr(resp.headers, 'getheader'):
                    resp.getheader = lambda x: resp.headers.getheader(x)
                elif hasattr(resp.headers, 'get'):
                    resp.getheader = lambda x: resp.headers.get(x)

        data = resp.data.decode('UTF-8')
        try:
            if 300 <= statuscode < 304:
                return self._request('GET', resp.getheader('location'))
            if 399 < statuscode <= 500:
                return {'status': statuscode, 'value': data}
            content_type = []
            if resp.getheader('Content-Type') is not None:
                content_type = resp.getheader('Content-Type').split(';')
            if not any([x.startswith('image/png') for x in content_type]):

                try:
                    data = utils.load_json(data.strip())
                except ValueError:
                    if 199 < statuscode < 300:
                        status = ErrorCode.SUCCESS
                    else:
                        status = ErrorCode.UNKNOWN_ERROR
                    return {'status': status, 'value': data.strip()}

                # Some of the drivers incorrectly return a response
                # with no 'value' field when they should return null.
                if 'value' not in data:
                    data['value'] = None
                return data
            else:
                data = {'status': 0, 'value': data}
                return data
        finally:
            LOGGER.debug("Finished Request")
            resp.close()

From the above realization, execute is sending requests to remote servers; the invokes in execute are The request method sends the http request and returns the relevant results, which are responded by the browser.

The principle of the request is described in the official note:

At its minimum, WebDriver talks to a browser through a driver.
Communication is two way: WebDriver passes commands to the browser through the driver, and receives information back via the same route.

The driver is specific to the browser, such as ChromeDriver for Google's Chrome/Chromium, GeckoDriver for Mozilla's Firefox, etc. Thedriver runs on the same system as the browser. This may, or may not be, the same system where the tests themselves are executing.
This simple example above is direct communication. Communication to the browser may also be remote communication through Selenium Server or RemoteWebDriver. RemoteWebDriver runs on the same system as the driver and the browser.

In a word, we use webdriver to talk with the browser and the browser will respond.

It is known from the above example that sending a request to the remote server by execute will interact with the browser through webdriver, and sending the defined command constant can obtain some relevant information.

Because in the code, our instance is webdriver instance, go to webdriver base class( selenium.webdriver.remote.webdriver) to query related information, and check whether there are related functions to obtain information. The following functions were found:

def title(self):
    """Returns the title of the current page.

    :Usage:
        title = driver.title
    """
    resp = self.execute(Command.GET_TITLE)
    return resp['value'] if resp['value'] is not None else ""
@property
def current_url(self):
    """
    Gets the URL of the current page.

    :Usage:
        driver.current_url
    """
    return self.execute(Command.GET_CURRENT_URL)['value']
@property
def page_source(self):
    """
    Gets the source of the current page.

    :Usage:
        driver.page_source
    """
    return self.execute(Command.GET_PAGE_SOURCE)['value']

The above is not a complete list. We simply try to use the above functions. The use methods are explained in the functions. Try to get title, current_url (current url), page_source:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.csdn.net")
print(driver.title)
print(driver.current_url)
print('Author blog: https://blog.csdn.net/A757291228')
#Original support, reprint please post the original link
# print(driver.page_source)

Results the page title and current web address were obtained successfully:

Try page_source:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.csdn.net")
print(driver.title)
print(driver.current_url)
print('Author blog: https://blog.csdn.net/A757291228')
#Support original, reprint please post link
print(driver.page_source)

Successfully obtained:

It's not easy to be original. Let's see how to support it! thank you