Posted on by & filed under django, python.

The longer I’m a programmer, the lazier I become. Several years ago I’d have been a giddy schoolgirl if you told me to write a templating engine from scratch. Or authentication, wow—Dealing with HTTP headers and sessions got me so excited!

Nowadays I wonder why things just can’t just work.

At Safari, there are lots of services with moving parts that need to be scheduled and I’ve gradually started to really dislike cron. Sure it’s great for one-off tasks, but handling lots of tasks asynchronously is not one of its strong suits. And really, I’m just too lazy to write the logic to handle failures, redos, and other catch-22’s that happen in the pipeline. Instead, I now use a combination of Django and the task queue Celery.

Enter Celery and Supervisor *on Ubuntu

Ubuntu is quite nice to work with, as they keep packages relatively up to date. Supervisor? Redis? They just work, almost like magic. Here’s the steps to get a cron-free world and running in a jif (with a Python virtual environment):

First, let’s install the necessary Ubuntu packages, create a working environment for the project, and get the necessary Python libraries. Let’s call the project Thing.

Now we can start to put the Django pieces together. Start a new Django project called thing with an app called automate where we’ll put our tasks. Also, add a serverconf/ directory to keep your server/service configs separate.

Your project should look something like this:

Add automate to the INSTALLED_APPS section in your and be sure to alter your DATABASES to use your backend of choice. My DATABASES looks like this:

Something to Do

Now let’s just create a basic framework that does something, like crawl a web site for content. Modify your automate/ to look like this:

import urllib2

from django.db import models

class WebContent(models.Model):
    # I like timestamps
    timestamp_created = models.DateTimeField(auto_now_add=True)
    timestamp_updated = models.DateTimeField(auto_now=True)

    url = models.CharField(max_length=255)
    content = models.TextField(null=True)

    def update_content(self):
        self.content = urllib2.urlopen(self.url).read()

Test it out, it should work just fine:

A Real, Grown-up, Cron-like Task

Now we need to start adding the ingredients to turn this into a celery task (the equivalent of a cronjob). First, add djcelery to your list of INSTALLED_APPS and remember to (thing)$ syncdb as well. Somewhere near the bottom of your thing/, add this:

import djcelery

from celery.schedules import crontab


BROKER_URL = "redis://localhost:6379/0"
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERYBEAT_SCHEDULE = {} # Will add tasks later

And while we’re at it, let’s modify the automate/ file, where celery tasks are actually defined:

from celery.task import task

from automate.models import WebContent

def update_all_sites():
    for rec in WebContent.objects.all():
       print "Updating site: %s" % rec.url

Test it out by running the celery daemon (aka worker). Then queue the task in a separate terminal.

1st terminal:

Note the following line to show that celery sees the task:

2nd terminal:

Your 1st terminal should have all kinds of awesome things going on:

Wow, it works! Now update your CELERYBEAT_SCHEDULE (like the timing in a cron job) in your to schedule the task.

    # Update web sites every 24h
    "update-web-sites": {
        "task": "automate.tasks.update_all_sites",
        "schedule": crontab(minute=0, hour=0),

The Final Piece

The final piece of the puzzle is to set up supervisor so that celery runs automagically alongside Django. Create a log directory called /var/log/thing. Your serverconf/thing-supervisor.conf should look something like this:

Finally, create the symlink so that your serverconf/thing-supervisor.conf is loaded when supervisor starts up:

There you have it, a complete install without using cron. Now you can go on to do all the cool things that celery supports, i.e. task retries, chaining, etc.


5 Responses to “Up and Running with Celery and Django (also cron is evil)”

  1. Andre


    you rock! that was exactly what I was looking for the last few days. I found the information on celery and brokers etc., but not as neatly put into one ready to go package as you did!


  2. Eduardo Silva

    Grate post, I was struggling with django and celery, and found this topic.
    Please note that your blog’s template seens broken and on the code part and where should appear ” (double quotes) we receive the html name: & quot;.
    Thank you!

  3. Dhiraj

    Do take note of the user setting in the supervisord conf file. I got a permission denied error because of it! wasted 15 min :(