Automating Web Scraping With Python and Cron

How to run cron job frequently 

Each entry in crontab begins with cron job frequency. The frequency or schedule contains five parts:

• Minute

• Hour (in 24-hour format)

• Day of the month

• Month

• Day of the week

The possible values are * (any value) or a number.

For example, if you want to run a task every hour, the schedule will be as follows:

0 * * * *

Notably, the cron process runs every minute and matches the current system time with this entry. In our case, it will only match when the system time is minute 0. All other fields have *, meaning these will fit for any value.

Since this task will run at 4:00, 5:00, 6:00, etc. -- effectively, the schedule will create a job run every hour.

Here are a few more examples.

• To run a task at 10 am on the 1st of every month, use the following:

0 10 1 * *

• To run a task at 2 pm (14:00) every Monday, type:

0 14 * * 1

Many sites, such as crontab.guru can help you build and validate a schedule.

How to remove python crontab job

To remove all crontab jobs, open the terminal and use this command:

crontab -r

If you want to remove a specific crontab job, you can edit the crontab file as follows:

crontab -e

Once in edit mode, you can delete the line for that job and save this file. The crontab will be configured with the updated contents, effectively deleting the cron job.

How to schedule Python script in crontab

First, decide what command you want to run. If you are not using a virtual environment, you can run your web scraping script as follows:

python3 /Users/upen/shopping/scraper.py

In some cases, you will have specific dependencies. If you're following recommended practices, it’s likely you've created a virtual environment.

Do note that it's often unnecessary to use source venv/bin/activate to release your venvo python with all its dependencies. For example, .venv/bin/python3 script.py already uses python3 from virtualenv.

A further recommendation would be to create a shell script and write the above lines in that script to make it more manageable. If you do that, the command to run your scraper would be:

sh /Users/upen/shopping/run_scraper.sh

The second step is to create a schedule. Let’s take an example of where the script must be run hourly. The cron schedule will be as follows:

0 * * * *

After finalizing these two pieces of information, open the terminal and enter the command:

crontab -e

Next, enter the following line, assuming you are using a shell script.

0 * * * * sh /Users/upen/shopping/run_scraper.sh

Upon saving the file, you may receive a prompt by your operating system, which will state your system settings are being modified.

Common reasons why crontab Python script isn't running

On macOS, the most common reason is cron’s lack of permission. To specify them, open System Preferences and click on Security & Privacy. In the Privacy tab, select Full Disk Access on the left and add the path of the cron executable. If you aren’t sure about the location of the cron executable, run the following command from the terminal:

which cron

Another common problem is that the system used Python 2 instead of 3 and vice versa. Remember that macOS and many other Linux distros ship with both Python 2 and Python 3. To fix this, find the complete path of the python executable file. Open the command prompt and run the following:

where python3

Take note of the python executable that you want to use. Unless you are using virtual environments, you must specify the complete path of the Python file.

Another common reason for failure is an incorrect path script. As a thumb rule, when working with cron, always use absolute paths.

Cron job vs SystemD vs Windows Task Scheduler vs AutoScraper

Cron is a tool specific to Unix-like operating systems such as macOS and Linux. Tools similar to it are Systemd (read as system-d) and Anacron. However, these are Linux-specific and aren't available on Windows.

For Windows, you can use the dedicated Windows Task Scheduler tool.

AutoScraper, on the other hand, is an open-source Python library that can work with most scenarios. You should note that the library isn’t meant to be an alternative to cron. It can automate the web scraping part, but you still have to write the Python script and use cron or one of the alternatives to run it automatically.

Scraper API Scheduler

Automating web scraping tasks is even easier with Oxylabs Scraper APIs. They come with a free feature called Scheduler, with which you can schedule multiple scraping and parsing jobs at any frequency. What’s great about it is that you can configure it to deliver the results straight to your preferred cloud storage and notify you once the delivery is done.

There are three simple steps to automating your tasks with Scheduler:

1. Submit a cron expression that tells us the frequency rate;

2. Specify the job parameters that we should execute;

3. Tell us when the schedule should be stopped.

You can find in-depth information on using Scheduler by visiting our documentation.

Conclusion

After having covered the crucial aspects of cron, crontab, and cron jobs, we hope you’ve gained a greater understanding of how web scraping automation is possible through above mentioned specific practices. As always, before automating web scraping projects, do make sure you conduct adequate research to determine which software and languages are the most relevant for your projects, as both Cron and Python have a variety of benefits and limitations when compared to the alternatives.

Alternatively, we offer advanced web scraping solutions for various cases, such as the Web Scraper API. With it, you get access to proxies, sophisticated scraping features, and a Scheduler to automate scraping and parsing jobs, so be sure to take a look.