How to Scrape Product Data From Wayfair: A Step-by-Step Guide

reCAPTCHA protection page

The reCAPTCHA protection page appears when Wayfair detects unusual browsing behavior, such as repeated or too-fast (for an organic user) navigation from page to page, indicating the use of automated scripts such as scrapers. The page looks similar to the one below:

Bypassing Wayfair scraping challenges

Elaborate anti-bot systems and an ever-changing web layout make automated data extraction difficult. As a consequence, when collecting data at scale from Wayfair, you might get blocked, banned, or blacklisted, not to mention constantly micromanaging the script to fix code breaks.

Wayfair is using Google’s reCAPTCHA service to block automated scrapers. It is an anti-bot protection service that uses fingerprinting algorithms and behavioral pattern recognition.

Oxylabs Wayfair API provides out-of-the-box support for bypassing the anti-bot measures by providing proxies, custom headers, user agents, and other features. This immensely eases the process and simplifies the scraper you would have to build.

Compared to regular scrapers, Wayfair Scraper API has multiple advantages, including:

• ML-driven proxy management

• Dynamic browser fingerprinting

• JavaScript rendering

How to Scrape Wayfair Product Data

Now, let’s see how to use Oxylabs Wayfair API to extract data from the Wayfair product page.

1. Set up the project environment by installing Python and required libraries

To begin scraping Wayfair data, prepare the project environment. If you already have Python installed, you can skip the Python installation and only install the dependencies in your active Python environment.

Installing Python

This tutorial is written using Python 3.11.2. However, it should also work with the older or latest version of Python 3. You can download the latest version of Python from the official web page.

Installing dependencies

Once you have downloaded and installed Python,  install the following dependencies by executing the command below in the terminal or command prompt:

python -m pip install requests bs4 pandas

This command will install the Requests, Beautiful Soup, and Pandas libraries. These modules will interact with the API and store data.

2. Fetch Wayfair product data using Wayfair Scraper API

Here’s a target product page. Use Wayfair Scraper API to fetch Wayfair product data and parse it using the Beautiful Soup library.

Signup for an Oxylabs account

To use the Oxylabs Wayfair API, create an Oxylabs account. You will find all the available APIs here. Make sure to use a 1-week free trial. You’ll have enough time to fine-tune the scraper. After the trial ends, you can keep using the service by seamlessly upgrading to a preferred plan.

Once you create your account, you will get your sub-user’s credentials, with which you can send network requests to the API.

Wayfair Scraper API overview

Before starting, let’s discuss some of the most useful query parameters of Wayfair Scraper API. The API operates in two modes.

Scraping using URL

Using this method,  you can scrape any Wayfair URL. You will only have to pass two required parameters: url and source. The source parameter should be set to wayfair, and the url should be a Wayfair web page URL.

It also takes optional parameters such as user_agent_type and callback_url. The user_agent_type tells the API which device the user agent will use (e.g., desktop). Lastly, the callback_url parameter is used to specify a URL to which the server should send a response after processing the request. Take a look at an example of a payload:

payload = {
    "source": "wayfair",
    "url": "https://www.wayfair.com/furniture/pdp/wade-logan-freetown-885-wide-reversible-sleeper-sofa-chaise-w010379019.html",
    "user_agent_type": "desktop",
    "callback_url": "<URL to your callback endpoint.>"
}

Scraping with query

The other method is to scrape data from search results. It also needs two parameters: source and query. This time, set the source to wayfair_search and put the search terms in the query parameter. This endpoint also supports additional parameters such as start_page, pages, limit, callback_url, and user_agent_type.

payload = {
'source': 'wayfair_search',
'query': 'sofa',
'start_page': 1,
'pages': 5,
'limit': 48
}

The result will start from the page number mentioned in the start_page parameter. You can retrieve several pages from the search result using the pages parameter and control how many search results per page to fetch using the limit parameter.

Sending network requests

To start writing your Wayfair scraper, import the libraries and create a payload with the necessary variables:

import requests
from bs4 import BeautifulSoup

product_url = "https://www.wayfair.com/furniture/pdp/wade-logan-freetown-885-wide-reversible-sleeper-sofa-chaise-w010379019.html"
payload = {
    "source": "wayfair",
    "url": product_url,
    "user_agent_type": "desktop",
}
username = "USERNAME"
password = "PASSWORD"

Notice username, password, and product_url variables. You will have to use your Oxylabs sub-user’s username and password. Also, if you wish, you can replace product url with the desired URL.

Next, send a POST request using the Requests module to Oxylabs' realtime API endpoint: https://realtime.oxylabs.io/v1/queries.

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=("USERNAME", "PASSWORD"),
    json=payload,
)
print(response.status_code)

In the code above, the POST method of the Requests module is used to send a POST request to the API. The sub-user’s credentials are passed for authentication, and the payload is sent in JSON format.

If you run this code, you’ll see 200 as an output which indicates success. If you get any other status code, recheck your credentials and payload.

3. Parse HTML data using BeautifulSoup

Now, you can parse the content of the JSON response. The JSON object will have the content of the webpage in HTML format. Use BeautifulSoup to parse the HTML from the response:

content = response.json()[“results”][0][“content”]
soup = BeautifulSoup(content, “html.parser”)

The default html.parser is in use. You can use a different parser if you want.

The soup object has the parsed HTML content. Now, parse the title, price, and rating from this object.

Title

Using a browser, inspect the HTML properties of the product title. To open the inspect tab, right-click on the product title and click inspect. You’ll see something similar to the image below: