Python Web Scraping Tutorial: Step-By-Step

Getting started in web scraping is simple except when it is not which is why you are here. Python is one of the easiest ways to get started as it is an object-oriented language. Python’s classes and objects are significantly easier to use than in any other language. Additionally, many libraries exist that make building a tool for web scraping in Python an absolute breeze.

In this web scraping Python tutorial, we will outline everything needed to get started with a simple application. It will acquire text-based data from page sources, store it into a file and sort the output according to set parameters. Options for more advanced features when using Python for web scraping will be outlined at the very end, with suggestions for implementation. By following the steps outlined below in this tutorial, you will be able to understand how to do web scraping. Yet, to save all the time and effort in building a custom scraper, we offer maintenance-free web intelligence solutions, such as our general-purpose Web Scraper API, so feel free to test it out with a free 1-week trial.

Try Web Scraper API now

This Python web scraping tutorial will work for all operating systems. There will be slight differences when installing either Python or development environments but not in anything else.

Building a web scraper: Python prepwork

Throughout this entire web scraping tutorial, Python 3.4+ version will be used. Specifically, we used 3.11 but any 3.4+ version should work just fine.

For Windows installations, when installing Python make sure to check “PATH installation”. PATH installation adds executables to the default Windows Command Prompt executable search. Windows will then recognize commands like pip or python without requiring users to point it to the directory of the executable (e.g. C:/tools/python/…/python.exe). If you have already installed Python but did not mark the checkbox, just rerun the installation and select modify. On the second screen select “Add to environment variables”.

Getting to the libraries

One of the Python advantages is a large selection of libraries for web scraping. These web scraping libraries are part of thousands of Python projects in existence – on PyPI alone, there are over 300,000 projects today. Notably, there are several types of Python web scraping libraries from which you can choose:

• Requests

• Beautiful Soup

• lxml

• Selenium

Requests library

Web scraping starts with sending HTTP requests, such as POST or GET, to a website’s server, which returns a response containing the needed data. However, standard Python HTTP libraries are difficult to use and, for effectiveness, require bulky lines of code, further compounding an already problematic issue.

Unlike other HTTP libraries, the Requests library simplifies the process of making such requests by reducing the lines of code, in effect making the code easier to understand and debug without impacting its effectiveness. The library can be installed from within the terminal using the pip command:

pip install requests

Requests library provides easy methods for sending HTTP GET and POST requests. For example, the function to send an HTTP Get request is aptly named get():

import requests
response = requests.get('https://oxylabs.io/')
print(response.text)

If there is a need for a form to be posted, it can be done easily using the post() method. The form data can sent as a dictionary as follows:

form_data = {'key1': 'value1', 'key2': 'value2'}
response = requests.post('https://oxylabs.io/', data=form_data)
print(response.text)

In general, proxy integration with requests library makes it very easy to use proxies that require authentication.

proxies={'http': 'http://user:password@pr.oxylabs.io:7777'}
response = requests.get('https://ip.oxylabs.io/location', proxies=proxies)
print(response.text)

But this library has a limitation in that it does not parse the extracted HTML data, i.e., it cannot convert the data into a more readable format for analysis. Also, it cannot be used to scrape websites that are written using purely JavaScript.

Beautiful Soup

Beautiful Soup is a Python library that works with a parser to extract data from HTML and can turn even invalid markup into a parse tree. However, this library is only designed for parsing and cannot request data from web servers in the form of HTML documents/files. For this reason, it is mostly used alongside the Python Requests Library. Note that Beautiful Soup makes it easy to query and navigate the HTML, but still requires a parser. The following example demonstrates the use of the html.parser module, which is part of the Python Standard Library.

#Part 1 – Get the HTML using Requests

import requests
url = 'https://oxylabs.io/blog'
response = requests.get(url)

#Part 2 – Find the element

from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, 'html.parser')
print(soup.title)

This will print the title element as follows:

<title>Oxylabs Blog | Oxylabs</title>

Due to its simple ways of navigating, searching and modifying the parse tree, Beautiful Soup is ideal even for beginners and usually saves developers hours of work. For example, to print all the blog titles from this page, the find_all() method can be used.

It may require you to use Developer Tools, an in-built function of web browsers that allows you to view the HTML of the page, and offers other functionalities for web developers. Open the Developer Tools by navigating to the browser settings, or use a keyboard shortcut: on Windows press F12 or Shift + Ctrl + I, and on macOS press Option + ⌘ + I. Then press the element selector button, which is in the top-left corner of the Developer Tools. Alternatively, you can press Shift + Ctrl + C on Windows, and Shift + ⌘ + C on macOS. Now, use the element selector to select a blog post title on the page and you should see the Developer Tools highlighting this element in the HTML source:

Looking at this snippet, you can see that the blog post title is stored within the <a> tag with an attribute class set to oxy-rmqaiq and e1dscegp1. 

Note: Since our website uses dynamic rendering, you may see the class set to css-rmqaiq and e1dscegp1. It is a good practice to print the whole HTML document in Python and double-check the elements and attributes in case you receive an empty response. 

Looking further, you should see that all the other titles are stored exactly the same way. As there are no other elements with the same class values throughout the HTML document, you can use the value e1dscegp1 to select all the elements that store the blog titles. This information can be supplied to the find_all function as follows:

blog_titles = soup.find_all('a', class_='e1dscegp1')
for title in blog_titles:
    print(title.text)
# Output:
# Prints all blog tiles on the page

Note that you must set the class by using the class_ keyword (with an underscore), otherwise, you will receive an error.

BeautifulSoup also makes it easy to work with CSS selectors. If a developer knows a CSS selector, there is no need to learn find() or find_all() methods. The following example uses the soup.select method:

blog_titles = soup.select('a.e1dscegp1')
for title in blog_titles:
    print(title.text)

While broken-HTML parsing is one of the main features of this library, it also offers numerous functions, including the fact that it can detect page encoding further increasing the accuracy of the data extracted from the HTML file.

What is more, it can be easily configured, with just a few lines of code, to extract any custom publicly available data or to identify specific data types. Our Beautiful Soup tutorial contains more on this and other configurations, as well as how this library works.

lxml

lxml is a parsing library. It is a fast, powerful, and easy-to-use library that works with both HTML and XML files. Additionally, lxml is ideal when extracting data from large datasets. However, unlike Beautiful Soup, this library is impacted by poorly designed HTML, making its parsing capabilities impeded.

The lxml library can be installed from the terminal using the pip command:

pip install lxml

This library contains a module html to work with HTML. However, the lxml library needs the HTML string first. This HTML string can be retrieved using the Requests library as discussed in the previous section. Once the HTML is available, the tree can be built using the fromstring method as follows:

# After response = requests.get() 
from lxml import html
tree = html.fromstring(response.text)

This tree object can now be queried using XPath. Continuing the example discussed in the previous section, to get the titles of the blogs, the XPath expression would be as follows:

//a[contains(@class, "e1dscegp1")]

The contains() function selects <a> elements only with a class value e1dscegp1. This XPath can be given to the tree.xpath() function. This will return all the elements matching this XPath:

blog_titles = tree.xpath('//a[contains(@class, "e1dscegp1")]')
for title in blog_titles:
    print(title.text)

Suppose you are looking to learn how to use this library and integrate it into your web scraping efforts or even gain more knowledge on top of your existing expertise. In that case, our detailed lxml tutorial is an excellent place to start.

Selenium

As stated, some websites are written using JavaScript, a language that allows developers to populate fields and menus dynamically. This creates a problem for Python libraries that can only extract data from static web pages. In fact, as stated, the Requests library is not an option when it comes to JavaScript. This is where Selenium web scraping comes in and thrives.

This Python web library is an open-source browser automation tool (web driver) that allows you to automate processes such as logging into a social media platform. Selenium is widely used for the execution of test cases or test scripts on web applications. Its strength during web scraping derives from its ability to initiate rendering web pages, just like any browser, by running JavaScript – standard web crawlers cannot run this programming language. Yet, it is now extensively used by developers.

Selenium requires three components:

• Web Browser – Supported browsers are Chrome, Edge, Firefox and Safari;

• Driver for the browser – As of Selenium 4.6, the drivers are installed automatically. However, if you encounter any issues, see this page for links to the drivers;

• The Selenium package.

The Selenium package can be installed from the terminal:

pip install selenium

After installation, the appropriate driver for the browser can be imported. Example for the Chrome browser as follows:

from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()

Now any page can be loaded in the browser using the get() method.

driver.get('https://oxylabs.io/blog')

Selenium allows use of CSS selectors and XPath to extract elements. The following example prints all the blog titles using a CSS selector:

blog_titles = driver.find_elements(By.CSS_SELECTOR, 'a.e1dscegp1')
for title in blog_titles:
    print(title.text)
driver.quit()  # closing the browser

Basically, by running JavaScript, Selenium deals with any content being displayed dynamically and subsequently makes the webpage’s content available for parsing by built-in methods or even Beautiful Soup. Moreover, it can mimic human behavior.

The only downside to using Selenium in web scraping is that it slows the process because it must first execute the JavaScript code for each page before making it available for parsing. As a result, it is unideal for large-scale data extraction. But if you wish to extract data at a lower-scale or the lack of speed is not a drawback, Selenium is a great choice.

Web scraping Python libraries compared

For this Python web scraping tutorial, we will be using three important libraries – BeautifulSoup v4, Pandas, and Selenium. Further steps in this guide assume a successful installation of these libraries. If you receive a “NameError: name * is not defined” it is likely that one of these installations has failed.

WebDrivers and browsers

Every web scraper, be it a general-purpose one or a SERP scraper, uses a browser as it needs to connect to the destination URL. For testing purposes we highly recommend using a regular browser (or not a headless one), especially for newcomers. Seeing how written code interacts with the application allows simple troubleshooting and debugging, and grants a better understanding of the entire process.

Headless browsers can be used later on as they are more efficient for complex tasks. Throughout this web scraping tutorial we will be using the Chrome web browser although the entire process is identical with Firefox.

Finding a cozy place for our Python web scraper

One final step needs to be taken before we can get to the programming part of this web scraping tutorial: using a good coding environment. There are many options, from a simple text editor, with which simply creating a *.py file and writing the code down directly is enough, to a fully-featured IDE (Integrated Development Environment).

If you already have Visual Studio Code installed, picking this IDE would be the simplest option. Otherwise, we highly recommend PyCharm for any newcomer as it has a very little entry barrier and an intuitive UI. We will assume that PyCharm is used for the rest of the web scraping tutorial.

In PyCharm, right click on the project area and New > Python File. Give it a nice name!

Importing and using libraries

Time to put all those modules you have installed previously to use:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver

PyCharm might display these imports in grey as it automatically marks unused libraries. Do not accept its suggestion to remove unused libs (at least yet).

You should begin by defining your browser. Depending on the webdriver you picked you should type in:

driver = webdriver.Chrome()

OR

driver = webdriver.Firefox()

Picking a URL

Before performing your first test run, choose a URL. As this web scraping tutorial is intended to create an elementary application, we highly recommended picking a simple target URL:

• Avoid data hidden in Javascript elements. These sometimes need to be triggered by performing specific actions in order to display the required data. Scraping data from Javascript elements requires more sophisticated use of Python and its logic.

• Avoid image scraping. Images can be downloaded directly with Selenium.

• Before conducting any scraping activities ensure that you are scraping public data, and are in no way breaching third-party rights. Also, do not forget to check the robots.txt file for guidance.

Select the landing page you want to visit and input the URL into the driver.get(‘URL’) parameter. Selenium requires that the connection protocol is provided. As such, it is always necessary to attach “http://” or “https://” to the URL.

driver.get('https://your.url/here?yes=brilliant')

Try doing a test run by clicking the green arrow at the bottom left or by right clicking the coding environment and selecting Run.

Even if no syntax or runtime errors appear when running our program, there still might be semantic errors. You should check whether we actually get the data assigned to the right object and move to the array correctly.

One of the simplest ways to check if the data you acquired during the previous steps is being collected correctly is to use print. Since arrays have many different values, a simple loop is often used to separate each entry to a separate line in the output:

for x in results:
   print(x)

Both print and for should be self-explanatory at this point. We are only initiating this loop for quick testing and debugging purposes. It is completely viable to print the results directly:

print(results)

So far your code should look like this:

driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
results = []
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')
for a in soup.find_all(attrs={'class': 'title'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)
for x in results:
    print(x)

Running your program now should display no errors and display acquired data in the debugger window. While print is great for testing purposes, it is not all that great for parsing and analyzing data.

You might have noticed that import pandas as pd is still grayed out so far. We will finally get to put the library to good use. Remove the print loop for now as you will be doing something similar by moving the data to a CSV file.

df = pd.DataFrame({'Names': results})
df.to_csv('names.csv', index=False, encoding='utf-8')

The two new statements rely on the pandas library. The first statement creates a variable df and turns its object into a two-dimensional data table. Names is the name of the column while results is the list to be printed out. Note that pandas can create multiple columns, you just do not have enough lists to utilize those parameters (yet).

The second statement moves the data of variable df to a specific file type (in this case CSV). The first parameter assigns a name to the soon-to-be file and an extension. Adding an extension is necessary as pandas will otherwise output a file without one and it will have to be changed manually. index can be used to assign specific starting numbers to columns. encoding is used to save data in a specific format. UTF-8 will be enough in almost all cases.

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
results = []
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')
for a in soup.find_all(attrs={'class': 'title'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)
df = pd.DataFrame({'Names': results})
df.to_csv('names.csv', index=False, encoding='utf-8')

No imports should now be grayed out and running the application should output a “names.csv” into your project directory.

Exporting the data to Excel

Pandas library features a function to export data to Excel. It makes it a lot easier to move data to an Excel file in one go. But it requires you to install the openpyxl library, which you can do in your terminal with the following command:

pip install openpyxl

Now, let's see how you can use Pandas to write data to an Excel file:

df = pd.DataFrame({'Names': results})
df.to_excel('names.xlsx', index=False, encoding='utf-8')

The new statement creates a DataFrame - a two-dimensional tabular data structure. The column label is Name, and the rows include data from the results array. Pandas can span more than one column, though that’s not required here as we only have a single column of data.

The second statement transforms the DataFrame into an Excel file (“.xlsx”). The first argument to the function specifies the filename - “names.xlsx”. Followed by the index argument set to false to avoid numbering the rows. Finally, the encoding is set to utf-8 to support a broader range of characters.

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')

results = []
for a in soup.find_all(attrs={'class': 'title'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)

df = pd.DataFrame({'Names': results})
df.to_excel('names.xlsx', index=False, encoding='utf-8')

To sum up, the code above creates a “names.xlsx” file with a Names column that includes all the data we have in the results array so far.

More lists. More!

Many web scraping operations will need to acquire several sets of data. For example, extracting just the titles of items listed on an e-commerce website will rarely be useful. In order to gather meaningful information and to draw conclusions from it at least two data points are needed.

For the purposes of this tutorial, we will try something slightly different. Since acquiring data from the same class would just mean appending to an additional list, we should attempt to extract data from a different class but, at the same time, maintain the structure of the table.

Obviously, you will need another list to store the data in:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
results = []
other_results = []
soup = BeautifulSoup(content, 'html.parser')
for b in soup.find_all(attrs={'class': 'otherclass'}):
# Assume that data is nested in ‘span’.
    name2 = b.find('span')
    other_results.append(name2.text)

Since you will be extracting an additional data point from a different part of the HTML, you will require an additional loop. If needed, you can also add another if statement to control the duplicate entries:

Finally, you need to change how the data table is formed:

df = pd.DataFrame({'Names': results, 'Categories': other_results})

So far the newest iteration of your code should look something like this:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
results = []
other_results = []
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')
for a in soup.find_all(attrs={'class': 'title'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)
for b in soup.find_all(attrs={'class': 'otherclass'}):
    name2 = b.find('span')
    if name2 not in other_results:
        other_results.append(name2.text)
df = pd.DataFrame({'Names': results, 'Categories': other_results})
df.to_csv('names.csv', index=False, encoding='utf-8')

If you are lucky, running this code will produce no error. In some cases pandas will output a “ValueError: arrays must all be the same length” message. Simply put, the length of the results and other_results lists is unequal, therefore pandas cannot create a two-dimensional table.

There are dozens of ways to resolve that error message. From padding the shortest list with “empty” values, to creating dictionaries, to creating two series and listing them out. We shall do the third option:

series1 = pd.Series(results, name='Names')
series2 = pd.Series(other_results, name='Categories')
df = pd.DataFrame({'Names': series1, 'Categories': series2})
df.to_csv('names.csv', index=False, encoding='utf-8')

Note that data will not be matched as the lists are of uneven length but creating two series is the easiest fix if two data points are needed. Your final code should look something like this:

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver


driver = webdriver.Chrome()
driver.get('https://your.url/here?yes=brilliant')
results = []
other_results = []
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')

for a in soup.find_all(attrs={'class': 'title'}):
    name = a.find('a')
    if name not in results:
        results.append(name.text)

for b in soup.find_all(attrs={'class': 'otherclass'}):
    name2 = b.find('span')
    if name2 not in other_results:
        other_results.append(name2.text)

series1 = pd.Series(results, name='Names')
series2 = pd.Series(other_results, name='Categories')
df = pd.DataFrame({'Names': series1, 'Categories': series2})
df.to_csv('names.csv', index=False, encoding='utf-8')

Running it should create a csv file named “names” with two columns of data.

Web scraping with Python best practices

Your first web scraper should now be fully functional. Of course, it is so basic and simplistic that performing any serious data acquisition would require significant upgrades. Before moving on to greener pastures, we highly recommend experimenting with some additional features:

• Create matched data extraction by creating a loop that would make lists of an even length.

• Scrape several URLs in one go. There are many ways to implement such a feature. One of the simplest options is to simply repeat the code above and change URLs each time. That would be quite boring. Build a loop and an array of URLs to visit.

• Another option is to create several arrays to store different sets of data and output it into one file with different rows. Scraping several different types of information at once is an important part of e-commerce data acquisition.

• Once a satisfactory web scraper is running, you no longer need to watch the browser perform its actions. Get headless versions of either Chrome or Firefox browsers and use those to reduce load times.

• Create a scraping pattern. Think of how a regular user would browse the internet and try to automate their actions. New libraries will definitely be needed. Use import time and from random import randint to create wait times between pages. Add scrollto() or use specific key inputs to move around the browser. It’s nearly impossible to list all of the possible options when it comes to creating a scraping pattern.

• Create a monitoring process. Data on certain websites might be time (or even user) sensitive. Try creating a long-lasting loop that rechecks certain URLs and scrapes data at set intervals. Ensure that your acquired data is always fresh.

• Make use of the Python Requests library. Requests is a powerful asset in any web scraping toolkit as it allows to optimize HTTP methods sent to servers.

• Finally, integrate proxies into your web scraper. Using location specific request sources allows you to acquire data that might otherwise be inaccessible.

If you enjoy video content more, watch our embedded, simplified version of the web scraping tutorial!