How to Bypass CAPTCHA With Selenium & Python

In this tutorial, you’ll learn how to handle CAPTCHA in Selenium and Python.  You’ll install and combine the undetected-chromedriver with Selenium to enable scraping complex websites with anti-bot protection such as Cloudflare, CAPTCHA, etc. Additionally, you’ll explore Oxylabs’ Web Unblocker, which will simplify your scraping scripts and increase scalability. Let’s get started. 

Bypass CAPTCHA with Selenium and Python

The first step is to install Python if you haven't installed it already. You can download it from the official website. Download the latest version or a version greater than 3.6; otherwise, undetected-chromedriver won’t work properly.

Step 1 - Install dependencies

Install the undetected-chromedriver and requests module. You can use the pip command given below:

pip install undetected-chromedriver requests

Step 2 - Import libraries

Now that you’ve installed undetected-chromedriver, you can import it as shown below:

import undetected_chromedriver as webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--use_subprocess")

browser = webdriver.Chrome(options=chrome_options)

Notice, you’ve also created a browser instance. This will open a Chrome window in the background in headless mode. 

Step 3 - Navigate to webpage

Use the browser instance to navigate to your target website. For this tutorial, let’s use https://sandbox.oxylabs.io/products as the target. 

browser.get("https://sandbox.oxylabs.io/products")

Step 4 - Take a screenshot

Take a screenshot to verify the page is loaded properly without showing any CAPTCHA or bot protection screen. You can use the save_screenshot method of Selenium.

browser.save_screenshot("screenshot.png")

Your screenshot might vary slightly due to screen size, but it’ll look similar to the one given below:

The page has loaded properly without showing any CAPTCHA and the undetected-chromedriver has rendered the Javascript files.

Key things to remember

• The undetected-chromedriver doesn’t hide your IP address. So, if your IP address has a low reputation score, you’ll get a CAPTCHA no matter what you do. 

• Some anti-bot protection services block headless browsers by default. In that case, you’ll have to set headless=False to avoid getting blocked.

• Without a rotating proxy, this method is unreliable for large-volume scraping. As the IP always remains the same, the anti-bot protection services can pick up the extra volume of requests from the particular IP address and block the requests by IP immediately.

• Configuring proxy with authentication is tricky with undetected-chromedriver as it uses the chromedriver under the hood. The chromedriver doesn’t support proxy authentication. However, you can install and use the seleniumwire extension to set up authenticated proxies with undetected-chromedriver.  

Bypass CAPTCHA with Web Unblocker

To perform large-scale web scraping while bypassing CAPTCHA, you’ll need a strong tool. Web Unblocker, an AI–powered proxy solution for bypassing IP blocks and CAPTCHAs, will automatically rotate proxies for you, so you don’t have to worry about manually managing a list of proxies for your bots.

Step 1 - Import libraries

Let’s use the requests module to set up Web Unblocker. 

import requests

Step 2 - Get Web Unblocker credentials 

Create an account to get the Web Unblocker credentials. Within a few clicks, you can sign up and get a 1-week free trial to develop and thoroughly test the solution.

Step 3 - Prepare Web Unblocker 

Web Unblocker’s host and port are unblock.oxylabs.io and 60000 respectively. Additionally, don’t forget to replace the USERNAME and PASSWORD with the correct credentials.

proxy = 'http://{}:{}@unblock.oxylabs.io:60000'.format("USERNAME", "PASSWORD")
proxies = {
    'http': proxy,
    'https': proxy
}

If you get any authentication-related errors in the later steps, don’t forget to check the Web Unblocker response codes here.

Step 4 - Fetch content

Now, you can use the proxies dict you created with the get method of the requests module. Web Unblocker also requires you to pass an extra parameter, verify=False, to the get method.  

page = "https://sandbox.oxylabs.io/products"
response = requests.get(page, proxies=proxies, verify=False)
print(response.status_code)
content = response.content

You should see the status code 200 if everything works as expected. The content of the page will be stored in the content object, which you can process later with HTML Parser libraries such as Beautiful Soup or parse using the Custom Parser. Web Unblocker also renders JavaScript for you, so you can use this method for dynamic websites as well. 

Conclusion

In this tutorial, we demonstrated how to handle CAPTCHA in Selenium and Python using undetected-chromedriver. We also explored Oxylabs’ Web Unblocker, which makes it more convenient to develop complex scrapers with the ease and comfort of simple code. 

Learn more about Selenium web scraping as well as see how to integrate it with Oxylabs’ proxies. And in case you’d like to perform public web data gathering from targets like Walmart and Amazon, check out our ready-to-use scraping solutions, such as Amazon Scraper API.