Scheduling Python Scripts With Crontab

alarm with coffee

Python is a versatile programming language that you can use for a wide range of projects from automating everyday tasks like organizing files that you’ve worked on by date to collecting data from external sensors. While the versatility allows users to take multiple approaches to the same program, it does not always guarantee the best results. Scheduling is one such problem. There are numerous different ways to schedule a python script to execute at a given time or a given time interval.

In fact, the first approach that I took to scheduling python scripts is using a while loop. I ran a while loop that runs forever and the script would constantly check to see if it is time for the main body of the script to be executed. To reiterate, this method **works**. The purpose of the program was to create a folder for the screen shots that I’ve taken that day and organize the screen shots in a chronological order.

The following is the gist of what I’ve wrote years ago when I was first learning to program, and under no circumstance should it be reproduced.

while True:
    now = datetime.datetime.now()
    if now.minute == 30 or now.minute == 0:
        if now.second == 0:
            do_something

I realize that this is a terrible way to do anything. It puts a ridiculous amount of strain on the system as it has to update the now variable and check if it matches the provided conditions continuously. The better way to achieve the same results would something like this.

while True:
    time.sleep(30 * 60) # sleep (wait) for 30 minutes
    do_something()

The code above uses the standard time library to wait for thirty minute to do something and it keeps on repeating. This method can work great under certain conditions. For example, if the machine on which you’re running the code is not doing anything else in particular or if the startup process outweighs the processing power it takes to run the body of the program, making the program wait for thirty minutes can actually be more efficient.

While both methods mentioned above get the job done, they both require python to be running in the background. If you’re looking for an alternative, this is where crontab steps in to save the day.

Cron

Cron is a linux daemon that can be used to execute scheduled commands (Linux Man-Pages). For regular linux users, crontab needs no introduction as it is one of the most often used utility. Cron is most frequently used to schedule regular backups or regular safety checks.

For example the following single line of cron performs a backup every day at two thirty in the morning.

30 2 * * * /home/user/scripts/backup.sh

The following is the general syntax for a cronjob.

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <command to execute>

from Wikipedia

As you can see from the syntax, you’re telling cron to execute the backup.sh file at two thirty. The * is a wild card that basically means every. For example, if you had a cronjob that starts with * * * * *, it would be telling the machine to do something every minute of every day. However, it must be noted that doing something every minute of all time is simply not practical and efficient unless you just want to annoy a friend.

DISCLAIMER: DON’T DO THAT TO A FRIEND. YOU WILL LOSE A FRIEND FOR SURE.

Python and Cron

As you have probably guessed from the title, a python program can be executed from the crontab.

1. Open the crontab

crontab -e

If your script does something that requires sudo permissions, instead of calling sudo crontab, you need to make sure that you list root as the user. You can do so by using the -u flag like the following.

crontab -e -u root

2. Editing the crontab

If it is your first time opening the crontab, then most systems will ask you to decide on your default editor. If you are familiar with Vim, feel free to use it, but if you are not familiar with Vim, choose Nano for your default editor to avoid having to shut down the machine to get out of Vim.

Once you have decided on your default editor, all you have to do now is schedule a cronjob. The syntax, as we’ve seen above, is simple. The ordering goes minute-hour-day-month-day_of_week followed by the task that you want to schedule. While the syntax itself is simple, writing bad cronjob can lead to not working at all or having unintended yet dire consequences. Luckily, there are numerous tools online like the crontab.guru that helps those who are new to cron write a functional cronjob.

Let’s look at an example. For my code, I want my RaspberryPi to collect and log the temperature and humidity data every thirty minutes. In this case, my cron would look something like code below.

30 * * * * BODY
00 * * * * BODY

Notice that I have two lines for “every thirty minutes.” This is because cron executes the command at the time indicated, not at that interval. This is a little piece of detail that is easy to miss, so make sure that your cron actually says what you want it to say.

Virtual Environments And Cron

When it comes to Python, like most languages, virtual environment is a crucial component. It allows you to have different versions of libraries and even multiple python versions as well. If the code that you want to execute requires specific versions of libraries in your virtual environments, then you have to let cron know in order to execute the program.

If you don’t know which python you’re running at the moment, you can ask the terminal. How? Well, it’s a little complicated. First, make sure that you’ve activated the appropriate virtual environment and open your terminal and type in the following.

which python3 # I'm using python3, but choose the appropriate version

I know, it’s crazy hard. The command above should return the path for your current python file. If the path is complicated and you prefer not typing the whole thing to avoid any errors, you can store it the path as an environment variable.

PYPATH=$(which python3)
echo $PYPATH # check if the path is stored correctly.

Cronjob Body

Now, all that’s left for us to do is put all of the pieces together. Let’s use what we’ve done so far to write the body.

There are two important parts of the body. The first is the file to be executed (or a command to be executed), and the second is telling where the outputs of the file or the command should be saved. First, the file that we want to run is a python file. Therefore, the first part of the body is exactly as we would expect.

python3 my_file.py

However, if you want to make extra clear that the correct python is executed, you would want to use the variable that we’ve configured earlier.

$(PYPATH) my_file.py

Here, I am assuming that the python file is located in the root folder. However, most likely, that will not be the case. Make sure that you include the full absolute path of the file that you want to run.

So far, my cronjobs look something like this.

30 * * * * $(PYPATH) my_file.py
00 * * * * $(PYPATH) my_file.py

If you do not need anything to be printed from the file, this is it! Thank you for reading. But! If you want something to be saved or logged after the file is executed, you want to designate a specific log file for each cronjob. This can be achieved through modifying the standard output.

Cron Log

If you save the cronjob as it is now, everything will work fine. However, if there is an error in running the command, it will be logged to the main cron log. In most cases, it will be easier to have separate log files for different cron jobs.

If you’re familiar with Linux, manipulating the output is a simple task. Even if you’re not, it’s not that hard. Let’s consider the cronjob below.

30 * * * * $(PYPATH) my_file.py >> pycron.log 2>&1

It can seem complicated, but let’s break this down.

• >> : The two angle brackets tell terminal to append the outputs of my_file.py to the end of pycron.log. It is important that you use two angle brackets instead of one, since using one will overwrite the existing file.
• pycron.log: This is where you want the standard output to be stored. I’ve named it pycron, but you can name it to be whatever you want.
• 2>&1: In Linux, 2 represents standard error, and 1 represents standard output. If something unexpected happens, 2 is where you would find relevant information. With 2>&1 you’re telling the system to print all of the related errors to the standard output for the file. In our case, that would be pycron.log. Be careful that you don’t forget the ampersand as the missing ampersand will make the errors to be stored to a file called 1.

Final Code

30 * * * * $(PYPATH) my_file.py >> pycron.log 2>&1
0 * * * * $(PYPATH) my_file.py >> pycron.log 2>&1

This is it. The two lines of cron shown above will run my python file every 30 minutes and print out any results or issues to the pycron log file.

Closing Remarks

So far, we’ve explored how to schedule python scripts using crontab. If the startup process is not overly expensive, using the crontab to schedule your scripts will be the most efficient way to run regular commands periodically. There are just so many interesting things that you can do with cron, so if you’ve found this article interesting or helpful, feel free to explore more on your own. If you find anything interesting, let me know!