We've released django-rq-cron, a cleverly-titled cron module sitting on top of RQ and Django RQ.
We use this to process all of our scheduled and periodic events: sending daily emails, running our checker system, and more. We're releasing it in hopes that other folks with the same set of concerns and characteristics might find it useful.
We're marking this initial release as 0.2.0 (a nice middle ground between "stable" and "totally experimental, do not touch") not because we don't think it's technically production-ready — it is. We have been using this in production for almost a year. But because I suspect the API in some of the interfaces that work for our idiosyncratic needs might not be broadly applicable.
The question when releasing one of these packages is, why build out a cron package in the first place?
The existing package we were using, django-cron, had some issues for us and rather than try to paper over the issues that were fairly systemic it made sense to spike out building our own system. That spike ended up turning out pretty well and over time we fleshed out something that felt fairly mature and robust and now here we are.
The specific issues we ran into were threefold:
- The first was that django-cron's core processing and modeling didn't quite meet our requirements. For instance, django-cron serially processes CronJobs, meaning that if you have ten crons all running and the fourth one fails, the sixth never even attempts to run. This is an issue that we know we could paper over, but felt like a bit of a core flaw relative to having each one independently run its own task without interrupting its temporal neighbors. There were also some one-off locking issues that totally made sense from a design point of view, but just didn't quite work from our perspective. It was more important for us to be able to easily rerun and understand the state of a given job, as opposed to protect from reruns, because we generally design our crons with idempotence in mind.
- The second issue had nothing to do with django-cron. Buttondown sits on top of Heroku, and Heroku's default cron scheduler does not make any time guarantees. Unfortunately, this manifests in peculiar and dangerous ways—for instance, it might execute 9 minutes after we've scheduled it to run, causing django-cron to completely miss the entire batch of crons for a given window. The right approach here is to build a dedicated "dyno" for running the Cron full-time, which feels wasteful when we already have a process dedicated to doing exactly that.
- The third issue was that it was difficult to reconcile two separate systems for scheduled and periodic tasks. We've invested a lot in observability and tooling around RQ, and it felt painful to have to throw all that away for anything that was running on a cron, even if it looked very similar in all respects except for the harness actually handling the execution.
django-rq-cron started as an experiment for a few particularly sensitive crons, and it ended up working so well that we quickly migrated all of our crons to it. And here we are, releasing it to the world!
Even if you feel like your cron needs are completely solved, hopefully you find some of this interesting. The codebase itself is fairly compact and uncomplicated. We've got a set of models for governing the cron state machine, a registrar pattern for allowing you to register a cron in much the same way you might register a Django model admin, and a runner to actually invoke the crons and enqueue them as RQ jobs.
Please take a look at the codebase and let me know if you have any questions or hot takes or violent criticisms.