Web Scraping With RegEx

Collecting data using RegEx

In this tutorial, the RegEx scraping target is product titles and prices from a dummy website intended for training purposes.

Project requirements:

• The latest version of Python.

• Beautiful Soup 4 library to parse HTML.

• Requests library to make HTML requests.

Importing libraries

Let’s begin with creating a virtual environment for the project:

python3 -m venv scrapingdemo

Activate the newly created virtual environment (the example for Linux):

source ./scrapingdemo/bin/activate

Now, install the required Python modules.

Requests is a library responsible for sending requests to the websites on the internet and returning their response. To install Requests, enter the following:

pip install requests

Beautiful Soup is a module used to parse and extract data from the HTML response. To install Beautiful Soup, enter the following:

pip install beautifulsoup4

re is a built-in Python module responsible for working with Regular Expressions.

Next, create an empty Python file, for example, demo.py.

To import the required libraries, enter the following:

import requests
from bs4 import BeautifulSoup 
import re

Sending GET request

Use the Requests library to send a request to a web page from which you want to scrape the data. In this case, https://sandbox.oxylabs.io/products. To commence, enter the following:

page = requests.get('https://sandbox.oxylabs.io/products')

Selecting data

First, create a Beautiful Soup object and pass the page content received from your request during the initialization, including the parser type. As you’re working with an HTML code, select HTML.parser as the parser type.

Expression # 1

Finding the pattern

First, inspect the title of the product to find the pattern. You can see above that every title is present after the same class name in the <h4 class="title css-7u5e79 eag3qlw7">The Legend of Zelda: Ocarina of Time</h4> format.

Generating the expression

Then, create an expression that returns data inside the element tag by specifying "(.*?)".

The first expression is as follows:

re_titles = r'class="title css-7u5e79 eag3qlw7">(.*?)<\/h4>'

Expression # 2

Finding the pattern

First, inspect the price of the product. Every price is present in a div tag in the <div class="price-wrapper css-li4v8k eag3qlw4">91,99 €</div> format.

Generating the expression

Then, create an expression that returns data inside the div element.

The second expression is as follows:

re_prices = r'class="price-wrapper css-li4v8k eag3qlw4">(.*?)<\/div>'

To conclude, use the expressions with re.findall to find substrings matching the patterns. Lastly, save them in the data variables.

title = re.findall(re_titles, product_html)
price = re.findall(re_prices, product_html)
data.append((title, price))

Saving the output

To save the output, loop over the pairs for the titles and prices and write them to the output.txt file.

with open("output.txt", "w", encoding="utf-8") as f:
   for title, price in data:
       f.write(f"{title}\t{price}\n")