Django: Adding an intermediate model to a ManyToMany field without one
Django’s Many to Many relationships allow you to have a many-to-many relationship without an intermediate model — this is great for when you don’t need anything except foreign keys on the through table, and gives you an easy API to work with to get related objects. But what about when business needs change, and now you do need additional fields on the intermediate model — and you need to make the change without any downtime or loss of data?
Let’s say we’ve got the following models:
tags = models.ManyToManyField(Tag, db_table='course_tag', related_name='courses', blank=True)
There is a table in my database called
course_tag, but I don’t have a
CourseTag model, so I can’t add additional fields to it. Creating this model doesn’t actually require any changes to your database, but you need to let your Django app know that the model exists, so that future modifying migrations work as expected (and so that you can query
CourseTag objects directly).
Django has a util called
inspectdb that comes in handy here, since we want our first migration to change nothing about the database. I ran this and copied the intermediate model into my
models.py without changing anything about it other than removing
managed = False from
class Meta — that I want it to be managed is the whole point.
I also went ahead and changed the line that said
tags = models.ManyToManyField(Tag, db_table='course_tag', related_name='courses', blank=True) to read
tags = models.ManyToManyField('Tag', through='CourseTag'). This also doesn’t change anything about your database, but does impact the APIs that are available to you within the ORM.
At this point, run
python manage.py makemigrations.
If you run this migration normally, you’ll get an error that says something like
django.db.utils.ProgrammingError: relation "course_tag" already exists. You have a couple of options to deal with this:
- Run the migration with the
--fakeoption appended. This will not change anything about your database, but will let Django know that the migration has been “run” so that any future
AlterTableoperations will run successfully. The downside here, at least in my case, was that our automatic build failed, because the test suite runs migration files. Tests can of course be run with the
--nomigrationsoption, in which case this likely won’t be an issue for you, but if it is, move on to option 2. (If you ever need to re-create your database from scratch, you’ll run into problems with this option also, unless you migrate in stages and fake this migration every time)
- Alter your initial migration to include the information from the newly generated one. I went with this option so my build would pass. You have to be a little careful about order here to make sure things that rely on something else aren’t called prior to their existence. Your tests (assuming you’re not running them with
--nomigrations, because you’ve chosen this option), will let you know if you’ve gone awry here. For me, I changed the following things about my initial migration:
- Deleting the original
- Adding the
- Adding the new
AddFieldoperation to add
Coursesusing the correct through relationship
- Adding the
At this point, you’re finished with the transition, and you can add fields to your intermediate model the way you would usually add new fields — I ran another migration that added the new fields with
null=True, pushed the code to make sure all fields were being populated, and then had a final clean-up migration to remove
null=True from the fields that didn’t need it. All the rows in your original database table will automatically be available in the new
CourseTag model, since they use the same database table.
Keep in mind that there are some differences to the Django API depending on what type of many to many field you’re using, so you may need to update some of your database queries for related objects using the through table correctly — namely that you can no longer run a query on the
Tag table directly referencing
courses. A query that used to read:
would now read: