How to Scrape Google Shopping Results: A Step-by-Step Guide

In today’s competitive business environment, it’s hard to imagine a scenario where an e-commerce company or a retailer stays in demand without turning to web scraping. To shortly answer why, gathering accurate public data from thousands of targets worldwide is what gives them a chance to draw actionable insights and, eventually, present customers with the best deals.

This tutorial will demonstrate how you can scrape publicly-available data from Google Shopping hassle-free. In addition to the guide itself, we’ll shortly cover whether it’s legal to scrape Google Shopping and what difficulties you can encounter in the process.

What is Google Shopping?

Formerly known as Google Products Search, Google Products, and Froogle, Google Shopping is a service that allows users to browse, compare, and shop for products from different suppliers who have paid to be featured on the website. 

While giving consumers an opportunity to choose the best offers among thousands of brands, Google Shopping is also beneficial for retailers. When a user clicks on a product link, they are redirected to the vendor’s website for purchasing; thus, Google Shopping acts as a solution for businesses to advertise their products online.

More information on how Google Shopping works can be found here. 

Google Shopping results page structure overview

The data you get when browsing Google Shopping depends on some input parameters: Search, Product, and Price. Let's briefly discuss each of these parameters:

• Search: A list of the items on Google Shopping with information about each item, such as its ID, title, description, price, and availability.

• Product: Information on a single product's listing, details about other retailers selling it, and the costs at which it’s offered.

• Price: A list of all the product retailers along with the prices they offer and other details like delivery information, total costs, store name, etc. 

Search page 

The Google Shopping search results page lists all the relevant items available for the required product. The below screenshot highlights different attributes of a results page for the query “levis.” 

• Search bar: Allows a user to search for any product on Google Shopping. 

• List of products: Lists all the products and the details of the searched product. 

• Filters: Allows you to apply any filter to your search, for example, price range, color, style, etc. 

• Sorting options: This drop-down list enables you to sort your search on multiple attributes, for example, increasing price, decreasing price, popularity, etc.

• The list of products shows an individual product with the following product attributes: product name, price, name of the retailer or store, delivery Information.

Products page

When you select a specific item from the search page, you are directed to the Products page. This page contains detailed information about that particular product, such as its pictures, key features, product details, product reviews, retailers and prices information, and much more.

• Product name: Title of the product. 

• Product Highlights: Main features to have a quick product overview. 

• Product details: Detailed description of the product.

• Prices: List of different retailers and their prices. 

• Product reviews: Product rating and customer reviews. 

• Min and max prices: Product’s minimum to maximum pricing range sold by different sellers.

• General specifications: General information about the product. 

Pricing page

This page lists all the prices of different retailers’ products. It also shows if a store or retailer is a trusted one or not. Moreover, it gives information if the retailer has a Google Guarantee. 

• Product name: Name of the searched product.

• Rating: Overall rating of the product and number of reviews. 

• Prices from different stores: List of retailers, along with their offers, prices, and the link to visit their website to buy the product. 

• Filters: These filters can be applied to the retailers’ list. 

Is it legal to scrape Google Shopping results?

In general, web scraping is legal as long as you strictly follow all the regulations surrounding the public data you wish to gather. However, we still recommend seeking professional legal advice to rule out any possible risks.

If you wish to dive deeper into the topic of web scraping legality, check out our extensive blog post.

The pain of scraping Google Shopping

Though doable, scraping Google Shopping might not be the easiest task to take on. Not only is Google Shopping good at detecting automated requests, but it also requires parsing JavaScript, which is an “expensive” operation that slows down the scraping process. 

Therefore, to make sure you effortlessly scrape and parse a variety of Google Shopping page types, it’s best to rely on a high-quality scraping solution, such as Oxylabs’ Google Shopping API. This SERP API is specifically designed to deal with the challenges of Google scraping process and lets you gather accurate real-time data globally. If you want to extract data from the Google search engine, check out our other tutorial on how to scrape Google search results.

Step-by-step guide for scraping Google Shopping results using Google Shopping API

Step 1: Set up Python and install required libraries

To get started, you must have Python 3.6+ installed on your system. Then, you need to install the following packages to code the scraper. 

• Requests - to send the request to the API.

• Pandas - to populate the data in the DataFrame data structure. 

To install the packages, use the following command:

pip install requests pandas

Step 2: Set up a payload

Search page

The first step is creating a structure payload containing different query parameters. Below is a list of the query parameters and their brief description. 

For more detailed information on the parameters, check out our documentation.

Using the parameters mentioned in the table, we can create a payload structure as follows:

payload = {
   'source': 'google_shopping_search',
   'domain': 'com',
   'query': 'levis',
   'pages': 1,
   'context': [
       {'key': 'sort_by', 'value': 'pd'},
       {'key': 'min_price', 'value': 30},
   ],
   'parse': 'true',
}

Step 3: Send a POST request

After the payload structure is ready, you can create the request by passing your authentication key.

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

Step 4: Extract product data from a JSON response

We will be extracting Product Title, Price, and Store name from the response. Since we made the payload parameter parse: true, so we will get the JSON response. We can get all this data from the JSON response.

The code below extracts the data from JSON format and stores it in DataFrame.

#Get the content from the response
result=response.json()['results'][0]['content']
products = result['results']['organic']


#Create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Price', 'Store'])


#iterate through all the products
for p in products:
   title = p['title']
   price = p['price_str']
   store = p['merchant']['name']
   df = pd.concat([pd.DataFrame([[title, price, store]], columns=df.columns),
                   df], ignore_index=True)

The script extracts relevant product information from the response and stores it in the df DataFrame.

Step 5: Save extracted data to a CSV using Pandas

Using the following script, we can easily export the DataFrame to CSV or JSON files:

df.to_csv('google_shopping_search.csv', index=False)
df.to_json('google_shopping_search.json', orient='split', index=False)

Let’s put all the code together and see the output. 

import pandas as pd
import requests


# Structure payload
payload = {
   'source': 'google_shopping_search',
   'domain': 'com',
   'query': 'levis',
   'pages': 1,
   'context': [
       {'key': 'sort_by', 'value': 'pd'},
       {'key': 'min_price', 'value': 30},
   ],
   'parse': 'true',
}


# Get response
response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,


)


#Get the content from the response
result=response.json()['results'][0]['content']
products = result['results']['organic']


#Create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Price', 'Store'])


#iterate through all the products
for p in products:
   title = p['title']
   price = p['price_str']
   store = p['merchant']['name']
   df = pd.concat([pd.DataFrame([[title, price, store]], columns=df.columns),
                   df], ignore_index=True)


#Copy the DataFrame to CSV and JSON files
df.to_csv('google_shopping_search.csv', index=False)
df.to_json('google_shopping_search.json', orient='split', index=False)

The script doesn’t contain any print statements and writes everything in CSV and JSON files. Let’s look at a portion of the output CSV file.

As expected, the output CSV contains the Product Titles, Prices, and Store information for all the products listed on the search page.

Now, let’s scrape a specific product page. 

Product page

The payload structure will be created using different parameters for the products page. Below is a list of the query parameters and their brief description. 

Once again, for more detailed information on the parameters, check out our documentation.

We will be using product ID 4505166624001087642 for scraping. Using the parameters mentioned in the table, we can create a payload structure like this:

payload = {
  'source': 'google_shopping_product',
  'domain': 'com',
  'query': '4505166624001087642',
  'parse': 'true',
}

After the payload structure is ready, you can create the request by passing your authentication key. 

response = requests.request(
  'POST',
  'https://realtime.oxylabs.io/v1/queries',
  auth=('username', 'password'),
  json=payload,
)

We’ll extract the Product Title, Product Details, Highlights, Rating, and Reviews Count from the response received. Like in the previous section, we’ll use JSON response and extract our desired output. You can see the structure of JSON output here. 

# Get the content
product=response.json()['results'][0]['content']


# create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Product Details',
                          'Highlights', 'Rating', 'Reviews Count'])


# Get the elements from the response object
title = product['title']
details = product['description']
highlights = product['highlights']
rating = product['reviews']['rating']
reviews_count = product['reviews']['reviews_count']


# Add all the elements in DataFrame
df = pd.concat([pd.DataFrame([[title, details, highlights, rating, reviews_count]],
                           columns=df.columns), df], ignore_index=True)

In the above code, we’ve created a DataFrame object that will save all the extracted data in it. We can print this DataFrame or write it in CSV or JSON files. 

# Copy the data in CSV and JSON file
df.to_csv('google_shopping_product.csv', index=False)
df.to_json('google_shopping_product.json', orient = 'split', index = False)


# Print the data on screen
print ('Product Name: ' + title)
print ('Product Details: ' + details)
print ('Product Highlights: ' + str(highlights))
print ('Product Rating: ' + str(rating))
print ('Reviews Count: ' + str(reviews_count))

Let’s put all the code together and see the output. 

import pandas as pd
import requests


# Structure payload.
payload = {
  'source': 'google_shopping_product',
  'domain': 'com',
  'query': '4505166624001087642',
  'parse': 'true',
}


# Get response.
response = requests.request(
  'POST',
  'https://realtime.oxylabs.io/v1/queries',
  auth=('username', 'password'),
  json=payload,
)


# Get the content
product=response.json()['results'][0]['content']


# create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Product Details',
                          'Highlights', 'Rating', 'Reviews Count'])


# Get the elements from the response object
title = product['title']
details = product['description']
highlights = product['highlights']
rating = product['reviews']['rating']
reviews_count = product['reviews']['reviews_count']


# Add all the elements in DataFrame
df = pd.concat([pd.DataFrame([[title, details, highlights, rating, reviews_count]],
                           columns=df.columns), df], ignore_index=True)


# Copy the data in CSV and JSON file
df.to_csv('google_shopping_product.csv', index=False)
df.to_json('google_shopping_product.json', orient = 'split', index = False)


# Print the data on screen
print ('Product Name: ' + title)
print ('Product Details: ' + details)
print ('Product Highlights: ' + str(highlights))
print ('Product Rating: ' + str(rating))
print ('Reviews Count: ' + str(reviews_count))

We’ve just successfully scraped a product page at Google Shopping. Let’s move on to scrape the Pricing page.

Pricing page

The payload structure will be created using different parameters for the pricing page. Below is a list of the query parameters and their brief description. 

More information on the parameters can be found in our documentation.

We’ll be using product ID 4505166624001087642 for scraping. Using the parameters mentioned in the table, we can create a payload structure like this:

payload = {
   'source': 'google_shopping_pricing',
   'domain': 'com',
   'query': '4505166624001087642',
   'parse': 'true'
}

After the payload structure is ready, you can create the request by passing your authentication key. 

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

We’ll be extracting Product Name, Special Offer, Item Price, Total Price and Shipping charges from the JSON response received. You can find the structure of the JSON response here. 

result = response.json()['results'][0]['content']
title = result['title']
pricing = result['pricing']
# Create a DataFrame
df = pd.DataFrame(columns=['Product Name', 'Special Offer',
                          'Item Price', 'Total Price', 'Shipping'])


for p in pricing:
   offer = p['details']
   item_price = p['price']
   total_price = p['price_total']
   shipping = p['price_shipping']
   df = pd.concat([pd.DataFrame([[title, offer, item_price,
                                  total_price, shipping]], columns=df.columns), df],
                  ignore_index=True)

The above script stores the extracted data in a DataFrame object. Therefore, saving data in CSV, JSON, or other formats is easy. Just execute the following code to save the whole data in CSV and JSON files.

df.to_csv('google_shopping_pricing.csv', index=False)
df.to_json('google_shopping_pricing.json', orient='split', index=False)

Let’s put all the code together and see the output. 

import pandas as pd  # include the pandas library for DataFrame
import requests  # Include the requests library


# Structure payload.
payload = {
   'source': 'google_shopping_pricing',
   'domain': 'com',
   'query': '4505166624001087642',
   'parse': 'true'
}


# Get response.
response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)


# Get the content from the response
result = response.json()['results'][0]['content']
title = result['title']
pricing = result['pricing']
# Create a DataFrame
df = pd.DataFrame(columns=['Product Name', 'Special Offer',
                          'Item Price', 'Total Price', 'Shipping'])


for p in pricing:
   offer = p['details']
   item_price = p['price']
   total_price = p['price_total']
   shipping = p['price_shipping']
   df = pd.concat([pd.DataFrame([[title, offer, item_price,
                                  total_price, shipping]], columns=df.columns), df],
                  ignore_index=True)


# Copy the DataFrame to CSV and JSON files
df.to_csv('google_shopping_pricing.csv', index=False)
df.to_json('google_shopping_pricing.json', orient='split', index=False)