In today’s web-driven world, automation and testing are crucial for ensuring the quality and reliability of web applications. Selenium, a widely used automation framework, provides developers and testers with a suite of tools for automating web browsers. One essential aspect of web automation is accurately locating elements on a web page to interact with them. This is where the find_element() using XPath method in Selenium proves invaluable.
Section 1: Understanding XPath Locators
XPath (XML Path Language) is a language for navigating XML documents and is widely used in web scraping and automation. XPath expressions consist of nodes, axes, and predicates, allowing for precise element targeting. By understanding XPath, you gain a powerful tool for locating elements on a web page.
XPath offers several advantages as a locator strategy. It provides flexibility in specifying element location by traversing the document structure. With XPath, you can target elements based on various criteria such as tag name, class, id, attribute, or text content. This flexibility allows for highly specific element identification, even in complex web pages.
NOTICE: In 2022, following Selenium version 4.3.0, the find_element_by_xpath() method was officially removed, along with all the other find_element_by_* functions. All of their functionality has now been merged into a function called find_element().
The first parameter of this function controls what its behavior. For the case of xpath, we will pass in the first parameter as “xpath”.
element = driver.find_element("xpath", **)
Section 2: find_element_by_xpath() Method
The find_element(“xpath”) method is a fundamental feature of Selenium that enables developers to locate elements on a web page using XPath expressions. By harnessing XPath, this method provides a reliable and robust way to identify elements, surpassing other locator strategies like CSS selectors.
Compared to alternative methods, find_element(“xpath”) offers greater versatility and precision in element identification. XPath allows you to navigate the entire document structure, making it ideal for handling complex page structures or elements nested deep within the HTML hierarchy.
Section 2.1: Usage and Syntax
The find_element() method is a fundamental feature of Selenium that enables developers to locate elements on a web page using XPath expressions. Nothing in the function name actually implies it has anything to do with XPath. This function can actually search for elements using a variety of methods, not just XPath. To search using XPath, you need to pass the string “xpath” into the first parameter.
element = driver.find_element("xpath", xpath_expression)
Here, xpath_expression represents the XPath expression that specifies the element to be located.
Creating XPath expressions involves understanding the structure of the HTML document and utilizing XPath syntax to traverse and target specific elements. Let’s explore how XPath expressions are constructed by referencing a sample HTML document:
Consider the following HTML snippet:
<html>
<body>
<h1>Welcome to the Website</h1>
<div class="content">
<p>This is a paragraph.</p>
</div>
</body>
</html>In this example, we have a simple HTML document with a hierarchical structure. To access elements within this document using XPath expressions, we can utilize various techniques. We will discuss these various techniques one-by-one in the following sections, and learn how to create XPath expressions for them.
Section 2.2: Absolute Paths
Absolute paths start from the root node of the document and specify the entire path to the desired element. For example:
To select the <h1> element, we will use the following XPath expression.
/html/body/h1
Html elements inside an XPath expression are separated using the “/” character. When this expression is passed into the find_element_by_xpath() method, the last Html element in the XPath expression is the one that is “returned”.
Let’s try it out.
element = driver.find_element("xpath", '/html/body/h1')
print(element.get_attribute('outerHTML'))
<h1>Welcome to the Website</h1>As you can see from the output, we were able to successfully select, and print out the “H1” html element. Let’s continue trying to create more XPath expressions to access various kinds of elements.
To select the paragraph tag:
element = driver.find_element("xpath", '/html/body/div/p')
Although the above command will work successfully on the HTML content we currently have, there is a flaw in it, which will cause it to return the wrong HTML element in certain cases. We will discuss this flaw later in the article. Keep reading to find out!
Section 2.3: Relative Paths
Relative paths start from a specific element and traverse the document structure accordingly. We start relative paths with a ‘//’ which stands for a “deep nesting” approach. For example:
To select a <p> element, we will do the following:
element = driver.find_element("xpath", '//p')
That’s it. This will automatically “deeply” acquire the paragraph element it can find within the HTML content, no matter how deeply nested it is.
The find_element() method is only capable of finding a single html element. If there are multiple “matches” found, only the first one is found. For e.g. in the case of the above expression, if multiple paragraphs are found, only the first one will be returned. To return all of the found matches, we can use the find_elements() method instead. The only difference is the extra “s” in the name, and the fact that this returns a list of HTML elements, instead of a single one.
Let’s make some modifications into our HTML. We now have two paragraphs, instead of one.
html = """
<html>
<body>
<h1>Welcome to the Website</h1>
<div class="content">
<p>This is the first paragraph.</p>
<p>This is the second paragraph.</p>
</div>
</body>
</html>
"""Let’s try using the find_elements() method on this. We had to make some little changes here and there, because the return value is a list of elements, instead of a single element.
element = driver.find_element("xpath", '//p')
element.get_attribute('outerHTML')
<p>This is the first paragraph.</p>
<p>This is the second paragraph.</p>As expected, it only returned the first one it found. Let’s try the find_elements() method.
elements = driver.find_elements("xpath", '//p')
for element in elements:
print(element.get_attribute('outerHTML'))
<p>This is the first paragraph.</p>
<p>This is the second paragraph.</p>Relative Paths are far more common than Absolute paths due to their ease of use. Extra care however, must be taken to ensure we do not select the wrong element. Relative paths can be partially combined with “absolute” paths for greater control over which elements are returned.
For example:
elements = driver.find_elements("xpath", '//div/p')
This code will return all paragraph tags directly located within a “div” element. “directly” located means that the paragraphs tags must be direct children of the <div> element, else they won’t get detected.
You can make this expression even more fancy by doing this:
elements = driver.find_elements("xpath", '//div//p')
Now it will detect paragraphs tags no matter how deeply nested they are into a <div> element.
This is the power of XPath expressions.
Section 2.4: Predicates
Predicates enable filtering of elements based on specific conditions. Remember that “flaw” we discussed earlier? It’s time to address the problem.
We will make another small change in the HTML content we have been using so far. We are adding another <div>, with the class of “footer” after the <div> of class “content”. This new “footer” <div> also has a paragraph element.
html = """
<html>
<body>
<h1>Welcome to the Website</h1>
<div class="content">
<p>This is the first paragraph.</p>
<p>This is the second paragraph.</p>
</div>
<div class="footer">
<p>This is the third paragraph.</p>
</div>
</body>
</html>
"""Now here is the problem. What if we wanted to access the paragraph tag in this new <div>? How do we do this? If we use the following expression, it will return the wrong <div> to us.
element = driver.find_element("xpath", '//p')
Turning this into an absolute path will not resolve the issue either.
element = driver.find_element("xpath", '/html/body/div/p')
This is because it does not know which <div> to go into it. It will simply enter the first div it locates within the <body> element, and return the first paragraph element it finds.
So how to fix this issue?
If you look at the HTML source again, the two <div> elements have unique “class” attributes. We can use this “class” attribute to filter out the divs. Here is how we can do this.
element = driver.find_element("xpath", "//div[@class='footer']/p")
print(element.get_attribute('outerHTML'))
<p>This is the third paragraph.</p>Here are some other common examples.
To select the <li> elements containing the text “Item”:
//li[contains(text(),'Item')]
To select the <div> elements with an id attribute:
//div[@id]
There are other examples like this out there, due to the number of different attributes and properties that html elements have.
Section 2.5: Axes
Axes allow navigation through the document structure based on relationships between elements. For example:
To select the parent <div> of the <p> element:
//p/parent::div
To select the following sibling <div> of the <h1> element:
//h1/following-sibling::div
This marks the end of the Selenium find_element() with XPath Expressions Tutorial. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the tutorial content can be asked in the comments section below.