Every year we invest in our products to provide the best solutions to our clients, and this time we took it to another level. We gathered great minds and put hundreds of hours to create an advanced AI-powered proxy solution – Web Unblocker!
We have been testing our market innovation for a few months now. After finally making all ends meet, we forged it into something we believe will make many of our clients’ lives much easier.
What is Web Unblocker?
Web Unblocker is an AI-powered proxy solution that easily bypasses sophisticated anti-bot systems. It enables effortless web data extraction from even the most difficult sites without any delays or errors. Our solution is as customizable as any regular proxy, but at the same time, guarantees a much higher success rate. Custom headers and IP stickiness are both supported, alongside reusable cookies and POST requests.
Integrating Web Unblocker is easy. It uses a simple backconnect proxy (a single entry node) and is compatible with your existing code. If you’ve already used regular proxies for data scraping, all you have to do is type in a Web Unblocker entry node instead of your current one. For more integration details, check out our documentation.
Web Unblocker features
ML-driven proxy management
Web Unblocker will carefully select and rotate proxies as well as evaluate which proxy pools work the best on a specific site.
An advanced proxy management feature contributes to a high success rate with the lowest response time possible.
Dynamic browser fingerprinting
AI-powered algorithms automatically generate optimal fingerprints for each target by using browser-specific combinations of user-agents and headers.
An optimal fingerprint greatly reduces the chance of getting blocked and delivers a better success rate for each request made.
ML-powered response recognition
Web Unblocker will create an effective feedback loop between the web scraping results and the experimentation engine.
The response recognition feature will make sure the quality of the outcome is determined effectively.
Auto-retry functionality
Web Unblocker will automatically retry data extraction in order to achieve the highest possible success rate.
In the case of an unsuccessful request, the auto-retry system will attempt a variety of tactics to ensure that data extraction eventually succeeds.
JavaScript rendering
Web Unblocker can render dynamically loaded content on websites.
Users will no longer need to use headless browsers to render page content as Web Unblocker will be able to load all data through JavaScript calls before it's sent back.
Additional benefits of Web Unblocker:
Simple integration using standard proxy protocols
Easily switch to Web Unblocker using standard proxy protocols.
Getting started is as easy as typing in our single entry node.
Highly scalable and customizable
Supports high volume of requests by utilizing Oxylabs global 102M+ IP proxy infrastructure.
Tailor requests on a country, city, or coordinate level (195 countries).
Send your own cookies and headers, or let us take care of them.
The main differences between Web Unblocker and self-managed proxies
Basic request
Residential Proxies:
curl -v -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" "https://example.com"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com"Web Unblocker uses a different entry node and is accessible via a different port. It is necessary to use the “-k” flag, which lets cURL not verify our TLS certificate. Additionally, the Web Unblocker authentication string does not require the user to prepend “customer-”.
Geolocation (country-level)
Residential Proxies:
curl -v -x pr.oxylabs.io:7777 -U "customer-USERNAME-cc-DE:PASSWORD" "https://example.com"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H "X-Oxylabs-Geo-Location: Germany"Geolocation preference is sent in a header with Web Unblocker, instead of being inside the auth string (as in Residential Proxies).
Sticky IP (session)
Residential Proxies:
curl -v -x pr.oxylabs.io:7777 -U "customer-USERNAME-sessid-abcde12345:PASSWORD" "https://example.com"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H "X-Oxylabs-Session-Id: abcde12345"In both cases, the session ID is arbitrarily chosen by the customer. The session ID is sent in a header with Web Unblocker and in the auth string with Residential Proxies.
Headers
Residential Proxies:
curl -v -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" "https://example.com" -H "Some-Header: Value"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H "Some-Header: Value"Headers are sent exactly the same way with both products, apart from differences outlined in the previous examples.
Cookies
Residential Proxies:
curl -v -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" "https://example.com" -H "Cookie: NID=1234567890; 1P_JAR=0987654321"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H "Cookie: NID=1234567890; 1P_JAR=0987654321"Cookies are sent exactly the same way with both products, apart from differences outlined in the previous examples.
POST
Residential Proxies:
curl -X POST -v -x pr.oxylabs.io:7777 -U "customer-USERNAME:PASSWORD" "https://example.com" -d "@/path/to/file"Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -d "@/path/to/file"POST requests are sent exactly the same way with both products, apart from differences outlined in the previous examples.
What Web Unblocker has and self-managed proxies don’t
JavaScript rendering
Residential Proxies: N/A
Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H “X-Oxylabs-Render: html”Unlike Residential Proxies, Web Unblocker can render JavaScript before passing the result back. This feature may be useful in few cases:
a) with sites which don’t load any useful content without the use of JavaScript.
b) when the target site’s bot detection tool uses JavaScript rendering as a way to determine whether the site is being accessed by a bot.
Custom Success Status Codes
Residential Proxies: N/A
Web Unblocker:
curl -k -x unblock.oxylabs.io:60000 -U "USERNAME:PASSWORD" "https://example.com" -H "X-Oxylabs-Successful-Status-Codes: 500,501,502,503"Web Unblocker lets the user specify custom success status codes. In the example above, the query specifies that a response from the target website with a response code of 500, 501, 502 or 503 should be passed back to the user and is considered successful. This feature is exclusive to Web Unblocker and does not have an equivalent in Residential Proxies.
Conclusion
Looking from a business perspective, upgrading to Web Unblocker will help you save up on proxy maintenance and development. Handling CAPTCHAs, keeping up with website updates, and creating JavaScript rendering will cost you time, and accordingly – money.
Human resources, constant proxy management, and other technical maintenance will come out to be costly. Therefore, the shift from Residential Proxies or other self-managed proxies is an easy one, and the benefits are obvious.
If you have any further questions regarding Web Unblocker, contact your account manager or book a call with our team by clicking the “Contact sales” button!
About the author
Gabija Fatenaite
Lead Product Marketing Manager
Gabija Fatenaite is a Lead Product Marketing Manager at Oxylabs. Having grown up on video games and the internet, she grew to find the tech side of things more and more interesting over the years. So if you ever find yourself wanting to learn more about proxies (or video games), feel free to contact her - she’ll be more than happy to answer you.
All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.
Related articles
Get the latest news from data gathering world
Forget about complex web scraping processes
Choose Oxylabs' advanced web intelligence collection solutions to gather real-time public data hassle-free.
Scale up your business with Oxylabs®
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub