Debugging background tasks inside loops and transactions
I picked up a bug fix the other day that I was able to solve fairly quickly based on a hunch, but even after I’d solved it, I didn’t totally understand why. So I went hunting (because c’mon, the why is the fun part!)
The crux of the problem was this: We were doing an operation on a list of objects, the whole loop wrapped in a database transaction (more detail on defining your own transactions and using it as a context manager here), and then calling a background task for each of them. Users were reporting that the action that was supposed to be performed in the background task wasn’t happening for all of the users in the loop — only some of them. To make this even more fun, I was able to replicate the issue fairly consistently on production, but not at all locally. Oh, concurrency (jk, I actually do 💙 distributed systems).
Here’s a pared down version of the problem code:
from django.db import transactionwith transaction.atomic():
for user in users:
...
user.save()
my_background_task.delay(user_id=user.id)
The issue here lies in the transaction. If the task is enqueued and picked up before the transaction closes, it’s possible that the database object on which it’s operating doesn’t actually exist yet. This issue is documented here, and can be resolved as the docs point out — by adding the function to a list of operations to be run only once the transaction closes. Like so:
