Oxylabs’ 2020 Trend Report: 2017-2019 Case Studies

The time has come to publish the third part of Oxylabs’ 2020 trend report. In this article, we go over three of the biggest e-commerce and also web scraping events for a couple of consecutive years now. These events: Valentine’s Day, Mother’s day and the biggest event for anyone working in either brick and mortar or online commerce businesses – Black Friday. Read part 2 of our report Datacenter Proxies here!

We encourage you to download the full report which you can do simply by filling in your email in the form below:

Looking at both of these graphs above, in the case of Valentine’s Day, for example, even though the request growth in 2019 was significant during the time around the event, the spike in traffic was not as big. From this we can assume that in 2019 the clients, based on prior experience, optimized their data gathering procedures, requiring less data to extract relevant insights.

Nevertheless, we can see that both for Mother’s Day and Valentine’s Day, retail sites intensified their traffic to prepare for both events in 2019. An especially significant rise of both requests and traffic is seen before Black Friday, indicating the extensive preparations made beforehand.

Case study 1: Valentine’s Day

The first big event that hits the e-commerce market every year is Valentine’s day. Days prior to February 14th, e-commerce websites start preparing in advance by scraping relevant sources. 

For example, in 2018 on Valentine’s Day, clients increased their proxy resources by an average of 13.03%, but they also increased traffic per IP by 27.71%, which likely led to 12.14% higher block rate on average.

On Valentine’s Day 2019, Oxylabs e-commerce clients decided to increase their web scraping resources by just 6.79% and raise the traffic per IP by 26.61%, which might have led to 51.65% higher block rate. This block rate was more than 4 times higher than in 2018 – a consequence of an insufficient amount of datacenter proxies used prior to the event.  

Case study 2: Mother’s Day

The second huge e-commerce market event of the year – Mother’s Day. Looking at choices made both in 2018, and 2019 – the same mistakes were made as on Valentine’s Day. 

In 2018, on Mother’s Day, companies managed to even drop the block rate by 9.34% while still increasing their traffic per IP by 21.83%, most likely because they expanded their datacenter proxy resources by an average of 32.3%.

In 2019, however, our clients sent 72.86% more traffic per IP despite the fact that the data gathering resources were only increased by 14.37%, which probably led to a 25.41% higher block rate.

Case study 3: Black Friday

Last but not least, the biggest e-commerce event of the year – Black Friday. Every year this event keeps growing, but in 2019 it became the busiest day for online shoppers in the U.S., attracting over 93M people, thus taking the crown from Cyber Monday (±83M). 

Looking through the history of Oxylabs’ client web scraping trends, it is easy to conclude that the e-commerce market takes this event very seriously.

On Black Friday that took place on November 24th, 2017, Oxylabs’ clients generated 40.9% more traffic per IP than usual, which most likely resulted in the average block rate jumping from 3.91% to 14.89%. Prior to the event, the web scraping resources were increased by an average of 7.22%.

On November 23rd, Black Friday of 2018, preparations started earlier. Furthermore, before the event, our clients increased their scraping resources by 29.12% and spread their data collection operations through a couple of days. This way, on Black Friday, they generated 31.8% more traffic per IP than usual and raised the block rate by only 7.15%.
This brings us to November 29th, Black Friday of 2019. Companies started their web scraping operations a week prior to the main event by increasing their resourcesby12.67% and spreading out the traffic through that whole week. On Black Friday, the traffic per IP was only raised by9.7% and the block rate remained almost the same (+1.22%).

Lessons learned 

All in all, what is evident from the case studies analyzed above – to carry out successful data-gathering operations with datacenter proxies, businesses ought to prepare in advance. Not only is it essential to spread out more intense web scraping through multiple days leading up to the event, but, also, to prepare better by upscaling datacenter proxy resources so that the significant increase in traffic could be handled without issues.

Going further, we present graphs indicating the request and traffic volume forecasts for 2020. The estimates were made on the assumption of similar growth continuing into 2020.

Unsurprisingly, our Data Analysis Department predicts that similar growth will happen around significant events such as Valentine’s Day, Mother’s Day, and Black Friday. Assuming similar scraping behavior, we will see more requests before, during, and after Valentine’s Day. With Mother’s Day, it is likely that the request rate will once again grow after the event, whereas for Black Friday, it is safe to assume that e-commerce market players will once again prepare in advance and start their operations before the major event.

As was the case with the forecast for the number of requests for the year 2020 for both datacenter and residential proxies, we predict similar growth trends for traffic volume. The only exception can be predicted in the case of Valentine’s Day, where the traffic in 2019 was not as significant as with the other two events, and we believe the same will repeat in 2020.

Read the third part of our report “Case Studies” here!