Python Selenium web automation

The Python Selenium library can be thought as a web (and browser) automation library. If you’ve heard of Pyautogui, a library that can automate the movements of a mouse and keyboard, Selenium is the web equivalent of this.

Selenium is able to take the place of the user while accessing web pages. It essentially recreates and automates a browser and can carry out pretty any task that a regular human could.

Selenium has the ability to seek out elements in a web page and interact with them. Such as the ability to click on buttons or links in a web page. Selenium is a more advanced way of accessing the web page as compared to other web libraries like BeautifulSoup and requests.

Installing Selenium

Install selenium through the command prompt using the following statement.

pip install selenium

The installation method may vary a bit depending on your Environment. “pip” just happens to be the most commonly used for installing Python modules.

Web and Browser Automation with Selenium

In order to actually begin executing any selenium commands, we first need to create the browser object. Using this object, we can do almost anything to a web page. Selenium revolves around this browser object. This browser object is created differently depending on the browser you want to use. For Firefox, you need the Firefox() function, where as for Chrome, you need the Chrome() function.

To set this object up however, we need to setup the “browser drivers” first.

We will now discuss various common installation methods for Selenium, with varying pros and cons, and different browser types. Pick the one which you find most suitable.

Hard-coding the Web-driver

This technique involves manually downloading the web-driver for the browser you want, and using its file location in your selenium code.

First, we need to import webdriver. Don’t worry, it’s part of selenium, so you don’t have to download it separately.

from selenium import webdriver

Now to create the browser object. If you wish to use Chrome, you’ll have to download the chromedriver.exe and link your Chrome() function to it, as shown below. You can download it using this link to Chrome’s website.

browser_obj = webdriver.Chrome(executable_path=filename)

Make sure you download the correct webdriver, as there the Chromedriver for each version is different. You can check your Chrome version by opening the Menu (the three dots in the upper right corner) then navigate to help and click on the “About Google Chrome” option.

Using Webdriver Manager

Webdriver manager is a separate convenience library that is designed to make integrating a web-driver into Selenium an easy task, without having to manually download anything. This method is also very flexible, and takes care of all the version issues that you might face with selenium and web-drivers.

This is a strongly recommended option for you to pick.

The first thing you need to do is install it.

pip install webdriver-manager

Here is how to setup the webdriver manager for chrome.

# selenium 3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

# selenium 4
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))

For learning about how to setup the other drivers, visit their github page here.

Using Selenium with Firefox

Regardless of which browser you choose, the syntax remains the same, except for that one line above (where we called the Chrome function). Each browser however, has it’s separate quirks, which you’ll find out if you dig deep enough into selenium.

Simply the same steps for the FireFox browser that you did for the Chrome browser. There will be only two changes. First, instead of download the chrome-webdriver, we will install the Firefox webdriver. The second change, is that instead of webdriver.Chrome() we will do webdriver.Firefox().

Keep in mind though, you actually need to have the FireFox browser installed and setup to use it. The same goes for any other browser you plan on using.

Inspecting the Web page

Before you actually begin writing any code, the first thing to do is scope out your target web page. For instance, you’re looking to automate your Facebook login. In order to automate it, you need to locate and identify the email and login fields.

You cannot achieve this without first inspecting their HTML code. It’s actually very simple. Right click on the email field, and click the Inspect option. This will open up the respective HTML text in front you.

All you have to do is identify some attribute or text that you can use to locate it. See below to understand.

Selenium Automation – Example

We’ll be using this site, CodersLegacy as a place to test selenium. Our goal is to find and click the button shown below using selenium. First, let’s find and connect to this page using it’s URL.

import selenium
from selenium import webdriver

url = "https://coderslegacy.com/libraries-in-python/python-selenium/"
driver = webdriver.Chrome('C:\\Users\Default\chromedriver.exe')
driver.get(url)

If this page opened up in front of you in a new browser window, you were successful. Now that we’ve established a connection, we can begin automating tasks.

Click me

X = driver.find_element_by_link_text('Click me')
X.click()

Adding these two lines is our first automation example. The driver.find_element function looks for an element that has the text “Click me”. There are numerous ways we might do this, such as locating the class. However, there may be other buttons sharing the same class, hence we need something unique to this button.

Example 2

A common use of selenium is automating logins in the browser. Let’s try automating our Facebook login.

After inspecting the Facebook login page, I see we are in need of three elements. First, the email field, which has the name attribute “email“. Secondly, the password field, which has the name attribute, “pass“. Finally, the login button itself, which we can access using it’s id, “loginbutton“.

Here’s our code.

import selenium
from selenium import webdriver

url = "https://facebook.com"
driver = webdriver.Chrome('C:\\Users\Shado\source\chromedriver.exe')
driver.get(url)

email = driver.find_element_by_name("email")
email.send_keys("[email protected]")
passw = driver.find_element_by_name("pass")
passw.send_keys("psswrd")

X = driver.find_element_by_id('loginbutton')
X.click()

It’s up to you now to keep practicing, and trying to automate new things. Remember to scope out your target page first, identifying each element separately. Keep an open mind and be willing to learn new things.

List of Web Driver Methods

Below is a list of all Methods for the Web driver object and web element.

Method Name	Description
browser.find_element_by_class_name(`name`) browser.find_elements_by_class_name(`name`)	returns element(s) with the class name, `name`.
browser.find_element_by_css_selector(`selector`) browser.find_elements_by_css_selector(`selector`)	returns element(s) that match the CSS selector, `selector`.
browser.find_element_by_id(`id`) browser.find_elements_by_id(`id`)	returns element(s) with the id, `id`.
browser.find_element_by_link_text(`text`) browser.find_elements_by_link_text(`text`)	returns <a> element(s) that have the exact text as `text`.
browser.find_element_by_partial_link_text(`text`) browser.find_elements_by_partial_link_text(`text`)	returns <a> element(s) that contain the text, `text`. It’s basically a partial match.
browser.find_element_by_name(`name`) browser.find_elements_by_name(`name`)	returns element(s) with the name value of `name`.
browser.find_element_by_tag_name(`name`) browser.find_elements_by_tag_name(`name`)	returns element(s) with the tag_names matching `name`.

List of Web element Methods

Attribute or Method	Description
`tag_name`	Returns the tag name, such as <p> for a paragraph element.
`get_attribute(name)`	Retrieves the value of the attribute called `name`.
`text`	Returns the text within the element, like the text in a paragraph. <p> text </p>.
`clear()`	For text field or text area elements, clears the text typed into it.
`is_displayed()`	Returns `True` is element is visible on screen, other wise returns `False`.
`is_enabled()`	Used for Input element. Returns `True` is the element is Enabled, else returns `False`.
`is_selected()`	For checkbox or radio button elements, returns `True` if the element is selected; otherwise returns `False`.
`location`	Returns a dictionary with keys ‘x’ and ‘y’ for the position of the element on the web page.

Selenium Alternatives

Chances are that you might be interested in Selenium Alternatives which are better for “Web Scraping” instead of “Web Automation”. One such alternative that his highly popular and commonly used for Scraping, is “Scrapy”.

Refer to this article to learn more about the differences between Selenium and Scrapy, and which one you should use.

This marks the end of the Python Selenium – web automation Article. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the article can be asked in the comments section below.

Here’s a link back to the main Python Libraries section where you can learn about other great libraries.