Back to blog
How to Scrape Home Depot Data: Step-by-Step Guide
3cbf.jpg)
Augustas Pelakauskas
Home Depot is one of the largest home goods retailers in the US. With its high popularity and vast online catalog, this e-commerce website holds valuable data for business insights. Home Depot data can fuel market research, price comparison, or inventory management.
In this guide, you’ll learn how to collect public data from Home Depot at scale with the help of automation, including the following Home Depot product data:
Product names
Prices
Availability
Review scores
1. Connecting to the API
To collect data from Home Depot, pair Python with Home Depot Scraper API.
Let’s start by creating a Python file to hold code. Run this command in the terminal:
touch main.pyIn the file you just created, define the initial parameters needed for a connection to the API:
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)USERNAME and PASSWORD are needed for authentication – these are your API credentials. Next comes the payload, where you can configure various parameters for the request you’re sending.
To begin with the request, you’ll need source, URL, and geo_location. Source is the parameter that defines what scraper (depending on a scraping target) should be used for a provided URL. In this case, it should be universal_ecommerce. The URL is the link that you want to scrape.
Finally, the geo_location parameter determines the country (or city) you’ll be scraping data from. As Home Depot is only available in the US, set it to the United States. You can read more about all the custom parameters for requests in our documentation.
The request is now set up. Let’s send it and check the response with a simple print:
print(response.json())A successful response should look something like this:
2. Extracting product name
Having the connection ready, you can start configuring the scraper to extract specific data from a product page. You can do so with the help of the Custom Parser feature, which is a part of the API.
First off, let’s get the product name. You’ll need to find the CSS handler by inspecting the HTML of the product page:
Now, let’s modify the payload with the Custom Parser instructions:
import requests
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
'parse': 'true',
'render': 'html',
'parsing_instructions': {
'product_name': {
"_fns": [
{"_fn": "css", "_args": ["div.product-details__badge-title--wrapper"]},
{"_fn": "element_text"}
]
},
}
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)
print(response.json())Add some additional parameters to your payload if needed. Render with the value html makes some of the JavaScript render on the site and loads part of the needed data. Parse with the value true means that you’ll use Custom Parser, while parsing_instructions contains what the name suggests – the parsing instructions, where you can put the CSS found earlier.
If you check the response, you’ll find the product name:
3. Scraping product price
Next, let’s gather pricing data. Find the CSS for it from the product page:
As there are many different products with their prices on the page, take the unique main price wrapper element first and then narrow it down with the specific price div within:
As for the code, you already have most of the payload set up from the previous request. You only need to make some minor adjustments by adding the parsing instructions:
import requests
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
"parse": 'true',
'render':'html',
"parsing_instructions": {
"price": {
"_fns": [
{"_fn": "css", "_args": ["div.price-detailed__qty-limit-wrapper"]},
{"_fn": "css", "_args": ["div.price"]},
{"_fn": "element_text"}
]
},
}
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)
print(response.json())Let’s run the code and check the response:
4. Getting availability
Getting the availability will be a tad bit trickier, as it’s something that you have to derive logically from the information found on the product page. Let’s assume the product is available if you see the button to add it to your cart.
Let’s find the CSS for this button:
And now for the code:
import requests
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
"parse": 'true',
'render':'html',
"parsing_instructions": {
"availability": {
"_fns": [
{"_fn": "css", "_args": ["div.buybox__atc"]},
{"_fn": "element_text"}
]
},
}
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)
print(response.json())Fetch the element for the Add to Cart button and get its text. Now, you need to implement the logic to check if the response has content. If it does, the product is available, as you can add it to the cart. If not, the product isn’t available:
raw_scraped_data = response.json()
results = []
for result in raw_scraped_data["results"]:
r = {
"availability": result["content"]["availability"] is not None,
}
results.append(r)
print(results)Let’s look at the print:
5. Collecting review score
Finally, to get the review score, find its CSS:
The only thing left is writing the code to fetch the score:
import requests
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
"parse": 'true',
'render':'html',
"parsing_instructions": {
"review_score": {
"_fns": [
{"_fn": "css", "_args": ["div.ratings-reviews__accordion-subheader"]},
{"_fn": "element_text"}
]
}
}
}
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)
print(response.json())If you look at the print after running the code, you should see the review:
6. Saving results
Now you can merge all the separate data extraction functionalities and save it all to a file for later use:
import requests
import json
USERNAME = "yourUsername"
PASSWORD = "yourPassword"
# Structure payload.
payload = {
'source': 'universal_ecommerce',
'url': 'https://www.homedepot.com/p/DEWALT-ATOMIC-20-Volt-MAX-Cordless-Brushless-Compact-Drill-Impact-Combo-Kit-2-Tool-with-2-1-3Ah-Batteries-Charger-Bag-DCK278C2/308067489',
'geo_location': 'United States',
"parse": 'true',
'render':'html',
"parsing_instructions": {
"product_name": {
"_fns": [
{"_fn": "css", "_args": ["div.product-details__badge-title--wrapper"]},
{"_fn": "element_text"}
]
},
"price": {
"_fns": [
{"_fn": "css", "_args": ["div.price-detailed__qty-limit-wrapper"]},
{"_fn": "css", "_args": ["div.price"]},
{"_fn": "element_text"}
]
},
"availability": {
"_fns": [
{"_fn": "css", "_args": ["div.buybox__atc"]},
{"_fn": "element_text"}
]
},
"review_score": {
"_fns": [
{"_fn": "css", "_args": ["div.ratings-reviews__accordion-subheader"]},
{"_fn": "element_text"}
]
}
}
}
# Get response.
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=(USERNAME, PASSWORD),
json=payload,
)
raw_scraped_data = response.json()
results = []
for result in raw_scraped_data["results"]:
r = {
"product_name": result["content"]["product_name"],
"availability": result["content"]["availability"] is not None,
"price": result["content"]["price"],
"review_score": result["content"]["review_score"],
}
results.append(r)
json_file_path = 'data.json'
with open(json_file_path, 'w') as json_file:
json.dump(results, json_file, indent=4)Wrapping up
As results prove, pairing Python and E-Commerce Scraper API is a seamless way to collect Home Depot data and avoid common web scraping-associated challenges. Make sure to check our technical documentation for all the API parameters and variables mentioned in this tutorial.
Also, check how to extract data from other popular targets, such as Best Buy, Zillow, eBay, Walmart, YouTube, Google Trends, and many more on our blog.
If you have any questions regarding the tutorial, feel free to drop us a message at support@oxylabs.io or via the live chat on our homepage, and our scraping experts will get back to you in a timely manner.
About the author
Augustas Pelakauskas
Senior Copywriter
Augustas Pelakauskas is a Senior Copywriter at Oxylabs. Coming from an artistic background, he is deeply invested in various creative ventures - the most recent one being writing. After testing his abilities in the field of freelance journalism, he transitioned to tech content creation. When at ease, he enjoys sunny outdoors and active recreation. As it turns out, his bicycle is his fourth best friend.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Related articles
Get the latest news from data gathering world
Try Home Depot Scraper API
Choose Oxylabs' Home Depot Scraper API to gather real-time product data hassle-free.
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub