No-downtime deployments: Changing the signature of a background task

No-downtime deployments is one of my favorite things to think about. It requires thinking about inputs and outputs and the different states of your application and database and other infrastructure during the phases of a deployment. It requires you to be intentional about how you make changes, not just what changes you make.

As business needs change and features evolve, it is common to change the signature of a function. Usually this is fine — you can control what the input of a function is, and adjust the logic within the function accordingly. Not so simple when your function is a background task though!

The Problem

The problem here is best demonstrated by example. Lets say you have a task that currently looks like this (Syntax in the examples assumes you’re using celery with Django, but the concepts apply even if you’re not!):

@shared_task(ignore_result=True)
def do_something(arg_1):
    ...

and you now need it to take an additional parameter, so that it looks like this:

@shared_task(ignore_result=True)
def do_something(arg_1, arg_2):
    ...

The issue here is that background tasks are definitionally not evaluated at the time they are enqueued, so a task that was enqueued with only arg_1 passed in will throw an error if you’ve deployed your change between when it was enqueued and when it will be processed — by the time it’s picked up…