Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers
hello@oxylabs.io

English (EN)

English

中文

Log in

Home

features

headless browser

Headless Browser

  • Render JavaScript and extract data from complex web pages

  • Configure browser instructions to automate user interactions

  • Customize browser behavior to simulate an organic user

*Headless Browser is a feature of Scraper APIs and Web Unblocker

Headless Browser

JavaScript rendering

Some website content appears only when rendering JavaScript. When web scraping with Oxylabs Scraper APIs, you can use the Headless Browser feature to execute JavaScript and load more data onto the page. Headless Browser renders JavaScript-based pages with a single line of code to save time and resources for the main task – subsequent data analysis.

Custom browser instructions

Perform action sequences to complete a specific task by setting up custom browser instructions. Headless Browser performs mouse clicks, inputs text, scrolls pages, waits for elements to appear, and more. To automate data extraction tasks that require user interaction, you should:

  • Study web page layout by inspecting HTML elements

  • Identify interactive elements containing target data

  • Define browser instructions to interact with the elements and load required data

{
   "source": "universal",
   "url": "https://www.ebay.com/",
   "render": "html",
   "browser_instructions": [
      {
         "type": "input",
         "value": "pizza boxes",
         "selector": {
            "type": "xpath",
            "value": "//input[@class='gh-tb ui-autocomplete-input']"
         }
      },
      {
         "type": "click",
         "selector": {
            "type": "xpath",
            "value": "//input[@type='submit']"
         }
      },
      {
         "type": "wait",
         "wait_time_s": 5
      }
   ]
}

Web scraping with browser automation

Automate browser behavior by setting wait times and timeout periods for dynamic elements to load before web scraping. Headless Browser lets you focus on data analysis, leaving infrastructure management and web data extraction activities like Document Object Model (DOM) manipulations, JavaScript rendering, and JavaScript fingerprinting to us.

What is Headless Browser?

A feature of Oxylabs Scraper APIs and Web Unblocker, Headless Browser allows you to define website-specific interactions to load dynamic elements.

Step 1

The user submits browser instructions to Scraper API/Web Unblocker.

Step 2

Headless Browser, a feature of Scraper APIs or Web Unblocker, sends an HTTP(S) request to the target web page.

Step 3

Headless Browser interacts with the web page in a user-defined way to load necessary data.

Step 4

Extracted data in raw HTML or a structured JSON file is transferred to the user's storage.

Solutions featuring Headless Browser

Get a free trial of our Scraper APIs and Web Unblocker, including access to Headless Browser.

SERP Scraper API

Public SERP data collected from the biggest search engines. 

  • Reliable, real-time data

  • Coordinate-level data

  • Exceptional ad scraping rate

Best for:

SERP data analysis, keyword data collection, brand monitoring.

From $49/month

Start free trial

E-Commerce Scraper API

Pricing and product information from e-commerce sites.

  • Adaptive Parser

  • Most e-commerce websites

  • Structured JSON data

Best for:

Competitor analysis, pricing intelligence, product catalog mapping.

From $49/month

Start free trial

Web Scraper API

Real-time data from the majority of websites.

  • Customizable request parameters

  • Convenient delivery

  • JavaScript rendering

Best for:

Website change monitoring, fraud protection, travel fare monitoring.

From $49/month

Start free trial

Web Unblocker

Access public data from the most difficult sites.

  • Easily imitate real website users

  • Bypass advanced anti-bot systems

  • Appear as an organic user on sites

Best for:

Anti-bot and CAPTCHAs handling for even the most difficult websites.

From $75/month

Start free trial

Aivaras Steponavicius

Senior Account Manager @ Oxylabs

With Headless Browser, you can automate user interactions to load dynamic data for extraction. This feature saves time and resources as you don’t have to develop and maintain your own headless browser solution.

Evelina Ablacinska

Account Manager @ Oxylabs

Headless Browser greatly simplifies the whole process of data extraction with browser instructions, allowing our clients to quickly sequence actions for site interaction.

A word from our dedicated Account Managers

With certain Enterprise plans, you get your own Dedicated Account Manager.

Useful resources

Documentation

Step-by-step guides

Learn how to set up your browser instructions with our technical documentation.

Oxylabs GitHub

Writing browser instructions

Check a tutorial on GitHub and copy the code directly to Headless Browser.

Oxylabs blog

What is a headless browser?

Get familiar with headless browsers: uses, examples, and limitations.

Frequently asked questions

What is the difference between a headless browser and a normal browser?

A headless browser doesn’t have a graphical user interface and is controlled programmatically with user-inputted code, allowing much more control compared to a regular browser.

What are the common use cases for using Headless Browser?

The use cases focus on web automation for data extraction:

  • Execution of JavaScript on target websites for data extraction.

  • Scraping JavaScript-heavy website components without managing additional infrastructure.

  • Data extraction from web pages that require user interactions (loading, scrolling, typing text).

  • Workflow streamlining – automating wait times for elements or resources to load before concluding a scraping task.

What are the benefits of using a headless browser?

Compared to a regular browser, a headless browser usually gives much more control over web browser-server communication as you can customize all parts of web interaction. On top of that, ease of scalability and running in the background without a graphical user interface utilize fewer resources, making all processes faster.

How does Headless Browser handle JavaScript rendering?

Headless Browser reads websites’ HTML code and executes JavaScript code found within. Part of the JavaScript code may make Headless Browser initiate HTTP requests to fetch additional data. The additional data is then loaded onto the HTML DOM (Document Object Model) and, as a consequence, made visible to the user.

What is the best headless browser?

The Headless Browser feature is the best option if you use an Oxylabs solution like a Scraper API or Web Unblocker.

There are various options for headless browsers, depending on your use cases and familiarity with certain software.

Do I need to be a Scraper APIs or Web Unblocker client to use Headless Browser?

Yes, you should be an active customer of Oxylabs APIs to use the Headless Browser feature. 

What are the disadvantages of headless browsers?

Not having a graphical user interface could result in rendering differences. When debugging, the lack of visual representation makes it more difficult to spot website layout changes, as reading the HTML source is the only guidance you get.

Get the latest news from data gathering world

I'm interested

Scale up your business with Oxylabs®

GET IN TOUCH

Certified data centers and upstream providers

Connect with us

Company

Proxies

Advanced proxy solutions

Top locations

Resources

Scraper APIs

Innovation hub

Privacy Policy
KYC Policy
Vulnerability Disclosure Policy
Speak Up

oxylabs.io© 2024 All Rights Reserved