Custom Celery Tasks: including the enqueuing request_id to a task_id in Django

Adrienne Domingus
5 min readAug 22, 2019
https://unsplash.com/photos/ZnMTiwDYXao

Debugging background tasks can be challenging, given they’re not necessarily executed in the order they’re enqueued (assuming you have multiple workers processing them), may be retried if they fail the first time through, and have no direct link to the web request that enqueued them.

Including the id of the web request that enqueued them in the id of the task itself can help alleviate a lot of these debugging issues, and deobfuscate some of the “what happened, and why?” questions that may arise.

I’m using the celery library with Django — this implementation is specific to that combination.

The primary implementation outlined below was first written up in this Rover blog — I found this article super helpful, but still had a couple hurdles to figure out, so thought I’d go into more detail here! Specifically, things that weren’t covered in that blog post that I’ll cover here include:

  • What if you’re enqueuing your tasks via .delay instead of .apply_async?
  • Options for the id of background tasks that are themselves enqueued by another background task.
  • How to use the custom task class you’ve defined

High Level Strategy

--

--