In this article we want to learn about Web Automation with Python Selenium, so Python Selenium is a powerful and open source framework that allows developers to automate web browsers actions. Using Selenium you can interact with web elements, navigating through pages or extracting data, Selenium empowers users to automate repetitive tasks and save time. Selenium supports different web browsers like Chrome, Firefox, Safari and Edge.
To work the examples of this tutorial you need some requirements, first you should have installed Python in your system, after that we need to install Python Selenium and you can use pip for that like this.
|
1 |
pip install selenium |
Selenium requires a browser specific driver to interact with web browsers. Download the appropriate driver for your preferred browser like ChromeDriver for Chrome and add it to your system PATH, or you can directly copy the driver and add that where you have your Python code.
You can download the drivers from here.
Python Selenium Automating Simple Actions
Let’s start with a simple example of automating a Google search using Python Selenium:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
from selenium import webdriver from selenium.webdriver.common.keys import Keys # Launch Chrome browser driver = webdriver.Chrome() # Navigate to Google driver.get("https://www.google.com") # Find the search input field and enter a query search_input = driver.find_element("name", "q") search_input.send_keys("Web automation with Python Selenium") search_input.send_keys(Keys.RETURN) # Wait for the search results to load driver.implicitly_wait(5) # Extract the search results results = driver.find_elements("xpath", "//div[@class='r']/a") for result in results: print(result.get_attribute("href")) # Close the browser driver.quit() |
So in the above code first we have imported the required classes from selenium.
|
1 2 |
from selenium import webdriver from selenium.webdriver.common.keys import Keys |
After that we launched Chrome browser:
|
1 |
driver = webdriver.Chrome() |
Using this code we can navigate to Google:
|
1 |
driver.get("https://www.google.com") |
In here find_element() method is used to locate the search input field on the webpage by specifying its attribute name (name) as “q”. send_keys() method is used to enter the desired search query (Web automation with Python Selenium) into the search input field. and lastly Keys.RETURN simulates pressing the Enter key to submit the search.
|
1 2 3 |
search_input = driver.find_element("name", "q") search_input.send_keys("Web automation with Python Selenium") search_input.send_keys(Keys.RETURN) |
The implicitly_wait() method sets a waiting period of 5 seconds, and it allows the browser to wait for the search results to load before proceeding to the next step.
|
1 |
driver.implicitly_wait(5) |
The find_elements() method is used to locate multiple elements on the page that match the specified XPath expression (//div[@class=’r’]/a). In this case, it finds all the search result links. get_attribute() method extracts the href attribute from each link element, and the URLs are printed to the console.
|
1 2 3 |
results = driver.find_elements("xpath", "//div[@class='r']/a") for result in results: print(result.get_attribute("href")) |
So we can say that the above code shows a basic example of web automation using Python Selenium to perform a Google search and retrieve the search results.
Python Selenium Interacting with Dynamic Web Elements
Selenium allows us to interact with dynamic elements on web pages. For example, let’s automate logging into a website:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
from selenium import webdriver # Launch Chrome browser driver = webdriver.Chrome() # Navigate to the login page driver.get("https://demoqa.com/login") # Find the username and password input fields and enter credentials username_input = driver.find_element("css selector", "#userName") password_input = driver.find_element("css selector", "#password") username_input.send_keys("geekscoders") password_input.send_keys("Geekscoders@123") # Submit the form login_button = driver.find_element("xpath", "//*[@id='login']") login_button.click() # Close the browser driver.quit() |
These lines initialize the Chrome browser using Chrome WebDriver and open the login page of the target website (in this case, “https://demoqa.com/login”).
|
1 2 3 4 5 6 7 |
from selenium import webdriver # Launch Chrome browser driver = webdriver.Chrome() # Navigate to the login page driver.get("https://demoqa.com/login") |
Using find_element_by_id() method, we locate the username and password input fields on the page using their HTML id attributes. After that we use the send_keys() method to enter the desired username and password values.
|
1 2 3 4 5 |
username_input = driver.find_element("css selector", "#userName") password_input = driver.find_element("css selector", "#password") username_input.send_keys("geekscoders") password_input.send_keys("Geekscoders@123") |
For submiting the login form, we locate the login button using its XPath expression and the find_element_by_xpath() method. after that we simulate a button click using click() method.
|
1 2 |
login_button = driver.find_element("xpath", "//*[@id='login']") login_button.click() |
And lastly we close the browser and terminate the WebDriver session using the quit() method.
|
1 |
driver.quit() |
Python Selenium Handling Waits and Timeouts
Web pages often have elements that load asynchronously or require time to appear. Selenium provides different methods to handle waits and timeouts:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC # Launch Chrome browser driver = webdriver.Chrome() # Navigate to the page with dynamically loaded element driver.get("https://example.com") # Wait for the element to be visible wait = WebDriverWait(driver, 10) element = wait.until(EC.visibility_of_element_located((By.ID, "element_id"))) # Perform actions on the element element.click() # Close the browser driver.quit() |
