When managing data across different environments, there are a number of recurring processes that are essential to maintaining data integrity. For example, taking regular database backups, sending out data health reports, and monitoring systems for slow queries. Writing scripts to schedule these operations helps relieve the pressure of remembering to perform these tasks and ensures critical operations are happening consistently.
One of the most popular ways to schedule tasks is by using a crontab file. For example, you can set up a cron job to run your backup every day at 6:00 p.m., which ensures that your data will always be recoverable.
In this article we'll walk through how to schedule recurring tasks using cron jobs. We'll discuss crontab’s structure and syntax, real-life use cases, and some limitations. We'll also touch on some alternatives to cron.
Introduction to cron and crontab
Cron is a time-based job scheduler used on Unix-like operating systems. It executes recurring tasks at specified intervals (e.g., hourly, daily, weekly, yearly, and so on) using the cron daemon. A daemon is a program used to run background processes that perform system tasks; the cron daemon uses a crontab file to specify what tasks to run and when to run them on the system.
A cron job is a command run by the cron daemon at scheduled intervals.
A crontab, short for "cron table", is a file that contains a list of commands that get run using a cron command to schedule tasks.
Here’s a code snippet that shows a sample crontab task when you run
crontab -l in your terminal:
As you can imagine, crontab files can be used for a variety of practical purposes. Here are a few common use cases:
A company might schedule weekly database backups of the data they have collected during the week. Since running a full database backup can utilize a large amount of system resources—thus impacting performance—many organizations perform backups at night, when fewer people are using the system.
Batch operations occur when a computer performs batches of jobs in sequential order. They’re typically used for very large tasks that get broken down into a group of processes to maximize efficiency. Consider an online e-commerce store, which might have thousands of customers subscribed to get email notifications when certain items are back in stock. Using a cron job, you could schedule a batch update that sends 100 emails out to users on 10-minute intervals until every subscribed customer receives an in-stock update.
Regular system updates ensure that your system is up-to-date with the latest security patches. Most of these security patches are done monthly—you may have noticed monthly update prompts on your phone, tablet, or laptop. These monthly updates can be scheduled using cron.
Task scheduling with crontab files
Recurring tasks are scheduled to run at a specified time by adding them to a crontab file. In this section, we'll walk through how to implement cron jobs using crontab files.
Crontab syntax and structure
The commands in a crontab file are written using a specific structure, as shown in the following code snippet:
Here’s a breakdown of the cron command structure as displayed above:
- Minute: The value expected is between 0–59 minutes.
- Hour: The value expected is between 0–23, representing the 24 hours in a day.
- Day of the Month: The value expected is between 1–31, representing the 31 possible days a month.
- Month of the Year: The value expected is between 1–12, representing the 12 months of the year. This value can also be defined using the first three letters of the month, like JAN, FEB, NOV, etc.
- Day of the Week: The value expected is between 0–6, from Sunday (0) to Saturday (6). You can also use the first three letters of the day, like SUN, WED, etc.
Putting this all together, a cron command that says
0 5 6 2 * would run every year on February 6 at 5:00 a.m. UTC, regardless of the day of the week. Similarly, the cron command
0 0-7 * 1 0 would run every hour at minute 0 from midnight to 7 a.m. UTC on Sundays in January, regardless of the day of the month.
Additional cron components
There are a variety of symbols that can be used within cron commands that help to better define time-related details. Here’s are some additional specifications that can be included:
- Any value (*): Asterisks are used as wildcards, which match all possible characters (e.g., every minute, every hour, every day, etc.). For example,
0 5 * * *runs the task every day at 5:00 UTC and
* * * * *would run every minute all of the time.
- Multiple values (,): Commas are used to separate multiple items in a list. For example,
2,4,8 * * * * commandwill run the task on the second, fourth, and eighth minute. For example, 1:02 and 2:08 would match.
- Multiple ranges (-): Dashes are used to define a range of values. For example,
* * 2-5 * * commandruns the task between February and May.
- Interval repetition (/): Slashes are used to signify frequencies within a given time period. For example,
*/4in the month field (
* * * */4 * command) will run the task every four months. (Note: This would be virtually the same as writing
4,8,12on the month field.)
Crontab commands are used to list, edit, or remove cron jobs from a crontab file. Here's how you can use them:
List cron jobs
To view all the tasks scheduled on your local machine, run the command
crontab -l in your terminal. You should see output similar to the screenshot below, showing a list of all your scheduled tasks:
Edit crontab file
To edit a crontab file or create a new one, run the command
crontab -e. You will be redirected to an editor similar to the one shown in the screenshot below. You can then add your tasks (one per line), save, and then exit from the editor.
Remove crontab file
To remove the current crontab file, run the command
Edit another user’s crontab file
To edit other users’ crontab files, run the command
crontab -u <username>. (Note: running this command requires that you have system administrator privileges.)
Examples of cron jobs
Below are a few examples of schedules you can use for your cron tasks.
Scheduling tasks every minute
The following command allows you to schedule tasks to run every minute:
* * * * * /bin/bash -l -c 'cd /Users/florencenjeri/projects/slackbot && RAILS_ENV=production bundle exec rake notifications_schedule:create --silent >> log/cron.log 2>&1
The command in the example above runs a task in a Ruby on Rails rake file. It has three parts:
/bin/bash -l -c: the shell used to run the cron commands
cd /Users/florencenjeri/projects/slackbot && RAILS_ENV=production bundle exec rake notifications_schedule:create --silent: the path to the task, as well as the name of task,
notifications_schedule:create, to be scheduled
>> log/cron.log 2>&1: specifies where to log the output of your scheduled tasks
Scheduling tasks every 15 minutes
The following command allows you to schedule tasks to run every 15 minutes:
*/15 * * * * command
Scheduling tasks every two hours
The following command allows you to schedule a task to run every two hours:
0 */2 * * * command
Scheduling daily tasks
The following command allows you to schedule a task to run every day at midnight:
0 0 * * * command
Scheduling weekly tasks
The following command allows you to schedule a task to run every week on Sunday at midnight:
0 0 * * 0 command
Scheduling yearly tasks
The following command allows you to schedule a task to run at midnight on the first of January every year:
0 0 1 1 * command
Scheduling tasks between specific hours
Finally, the following command allows you to schedule a task to run once an hour (at minute 0) every day from midnight to 7:00 a.m.
0 0-7 * * * command
The limitations of cron
Despite cron being the most widely used job-scheduling command, it has some limitations that derail its efficiency as a job-scheduling mechanism. Some of these limitations include the following:
Difficult to maintain
When using cron, you have to build your own monitoring and alerting systems to make cron jobs usable. Most tasks scheduled using cron are crucial, such as performing database backups. If a server goes down, for example, you'll need to be able to quickly identify which of your jobs are no longer running. With cron, these checks need to be manually added as there's no support for failure handling or alerts when jobs have failed.
Lack of security
As an organization scales, hundreds of cron jobs created by several different people can proliferate across several servers. Not only is this tough to maintain, but it's also problematic from a security perspective. It's a good security practice to limit access to production servers as a company scales, but if several people need to maintain cron jobs they've written in the past, it may be harder to lock down prod access.
Cron doesn't provide a central place to see, update, and manage all the cron jobs you have running. Especially as your organization grows, it becomes very difficult to have visibility into how many and which jobs are running on which machines since cron doesn't easily surface that information.
Cron doesn't log by default so you have to explicitly add in logic to capture logs for every cron jobs that you have running. Doing this for every script can be time-consuming and makes it challenging to monitor and remediate failed jobs.
Using Airplane to schedule tasks
Airplane is a developer platform for building internal tools. Airplane has first-class support for features like permissions and audit logs that make it an easier-to-use and safer alternative to cron. Scheduling tasks using Airplane can help you overcome some of the limitations described above.
Some of the advantages of replacing cron jobs with scheduled tasks in Airplane include the following:
Audit logging: Airplane automatically logs all tasks. The Activity page shows a log of all events with event details, actors, and timestamps.
Failure handling: On the Activity page you'll also see the task's status which will show if tasks have succeeded or failed. You'll automatically receive notifications via Slack and email when tasks fail for increased security and easier system maintenance. This should also make debugging easier.
Security and access controls: Using Airplane, you can set granular permissions for who can view, request, execute, and be an admin on a task-by-task basis. You'll be able to create group-based access controls which are not supported with crontabs.
Easy-to-use UI: You can view, create, and edit tasks all in one place from the Schedules page. This will provide a quick view of the schedule, when it's expected to run, and who has scheduled it. Cron jobs only show the tasks themselves (not who scheduled them). Editing a scheduled task is also simple with Airplane. You can do this directly from the UI.
Airplane provides a number of benefits that make it a simpler and safer alternative to cron. Airplane tasks are serverless, come with audit logs, permissions, and notifications, and can be used for scheduled tasks or for one-off operations that need to be run on-demand rather than on a schedule (such as testing your REST API).
Author: Florence Njeri