Next Chapter for Real-Time Crawler: New Products With Dedicated Focus

New look

All Three Scraper APIs can now be identified by their updated logos matching the targets they cover:

Same integration methods

To keep it simple for our clients and all those who tried out our products before this update, we decided to keep the same integration and authentication methods, as well as the same request parameters. 

SERP Scraper API authentication method 

SERP Scraper API uses basic HTTP authentication that requires a username and a password. It’s one of the simplest ways to start using this tool. The code example below shows how to send a GET request to scrape data from a search engine using the Realtime method.* For more information, read SERP Scraper API quick start guide.

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "SEARCH_ENGINE_search", "domain": "com", "query": "shoes"}'

*For this example, you need to specify the exact source to find available sources. Please refer to the SERP Scraper API documentation.

E-Commerce Scraper API authentication method 

E-Commerce Scraper API employs basic HTTP authentication which requires a username and a password. This is the easiest way to get started with the tool. The code example below shows how you can send a GET request to books.toscrape.com using the Realtime delivery method. For more information, read E-Commerce Scraper API quick start guide or E-Commerce Scraper API documentation:

curl --user "USERNAME:PASSWORD" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal_ecommerce", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "geo_location": "United States", "parse": true, "parser_type": "ecommerce_product"}'

Web Scraper API authentication method 

Web Scraper API employs basic HTTP authentication which requires a username and a password. This is the easiest way to get started with the tool. The code example below shows how you can send a GET request to https://ip.oxylabs.io/location using the Realtime delivery method. For more information, read Web Scraper API quick start guide or Web Scraper API documentation:

curl --user "USERNAME:PASSWORD"'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "universal", "url": "https://ip.oxylabs.io/location"}'

Wrapping up

Some changes all over the web and communication will happen over some period of time. All of these changes should be completed by the end of this year.

If you have any questions or suggestions regarding Scraper APIs changes, please feel free to contact us either via your Dedicated Account Manager or hello@oxylabs.io.