How to Scrape Yelp Data: Tutorial

Yelp is not only a place to find your next best dinner place. With years of compiling crowd-sourced content, Yelp is the perfect place to look for business data that can provide insights into local economic trends. 

In this article, why do businesses scrape Yelp data and what are the benefits of it? We will also show you a solution that can efficiently extract Yelp data at any scale.

Why scrape Yelp?

As mentioned above, throughout years of operating, Yelp has created a unique dataset filled with business details that can be scraped and used for various purposes – from journalism and academic research to business operations. On the business side, you might want to consider Yelp data for such things as:

• Customer sentiment analysis

• Market research 

• Competitor analysis

• Location planning 

• Reputation management

What data can be scraped from Yelp?

There is a variety of business data that can be gathered from Yelp, but you first need to be familiar with the types of pages that you will be scraping – the search page and the business page.

The search page will provide you with an overview of all the local businesses that fit your search criteria. This means that you will be able to collect such data as:

• Business name

• URL

• Reviews count

• Rating

• Tags

The business page, on the other hand, offers more detailed information about one specific business. This means that you get the information from the search page plus a more in-depth look at such things as reviews. In a nutshell, you can scrape:

• Business name

• URL

• Extended review information

• Contact information

• Opening times

• Amenities information 

So, Yelp offers a variety of data that can be used to uncover excellent business opportunities.

Project setup

For this tutorial, you’ll be using Python, so please download the latest version from their official website if you don’t have it already.

1. Installing dependencies

Next, you’ll have to install some libraries. All these libraries are available in the Python Package Index, so you can install them using a single command given below:

pip install requests bs4 pandas

2. Importing libraries

Now, import all the freshly installed libraries:

from bs4 import BeautifulSoup
import pandas as pd
import requests

As you can see, all three libraries are imported, and ready to use. The requests module will help you to send network requests. Once the server responds to the network requests, the BeautifulSoup module will be used for parsing the HTML content from the response object. And, pandas library will convert the parsed data into a CSV file.

3. Setting up Web Scraper API

To make things easier, we will use Oxylabs’ Web Scraper API which allows users to extract data from any website. Its main pros are that it has a built-in proxy rotator, custom device type, JavaScript rendering, etc. This means that there is significantly less chance that your scraping operations will encounter IP blocks or CAPTHCAs. 

 Let’s quickly look at the various parameters available to you.

Also, check out the complete list of parameters here. 

To use the Oxylabs Web Scraper API, you’ll need an Oxylabs account. Use your API user credentials and prepare a payload. The code will be similar to the one below:

page = "https://www.yelp.com/biz/memento-sf-san-francisco-3"
payload = {
    "source": "universal",
    "render": "html",
    "user_agent_type": "desktop",
    "url": page,
}
credentials = ("USERNAME", "PASSWORD")

4. Scraping Yelp business page

Once the payload is ready with the page url, you must send a POST request to the Web Scraper API. 

Don’t forget to pass the authentication credentials.

page = "https://www.yelp.com/biz/memento-sf-san-francisco-3"
payload = {
    "source": "universal",
    "render": "html",
    "user_agent_type": "desktop",
    "url": page,
}
credentials = ("USERNAME", "PASSWORD")

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=credentials,
    json=payload,
)
print(response.status_code)

If everything works as expected, you should see a response code 200. If you get any other response code then please see the documentation.

Inspecting elements

Before you start parsing the business page, let’s open it using a Web Browser and use the developer tool to identify the necessary CSS Selectors. You can right-click and select Inspect or just press CTRL + SHIFT + I (Windows) or ⌥ + ⌘ + I (macOS) to open developer tools. It should look like that:

Next, let’s find the CSS selectors for the name, reviews, rating, working hours and location elements.

As you can see, the name is available in an <h1> tag. Similarly, the rating is available in a span element with class css-1fdy0l5:

The review is available in an <a> tag with class css-19v1rkv:

And the address element has an <address> tag:

Last but not least, the working hours are wrapped in a <table> element:

Now, you can inspect each of the elements and find the CSS selectors for name, review count, rating, neighborhood and URL. For example, the name is wrapped in an <a> tag which is enclosed with an <h3> tag:

Similarly, you can find the rest of the CSS selectors using the developer tool. For your convenience, all of them are given below