A few months ago we published a series of emails about breaking up monolithic Django apps. The core of one of those emails was about moving Django models[0].
The issue with moving Django model classes is that Django’s migration system ties migrations to models as components of individual Django apps. This means that if you move a model out of one app and into another, it’s as good as deleting the model as far as the migrations system is concerned.
That’s bad news if you have production data.
If you move a model from app A to app B the migrations that Django generates will drop the model’s table even if you keep the table name consistent by using the Meta option “db_table” to name it explicitly.
The original suggestion in the email was to run the migration with the “fake” flag[1] _or _to modify the migration classes used for compatibility with automated deployment processes.
For these situations you can streamline the migration handling by _neutering _the operations (using customized migration operation classes) or delegating these migrations to another migration or management command run on deploy.
Never explained any further at the time, but the solution itself is quite simple, once you see it at least.
So here we go. First the technical explanation, and then the strategy.
Note: a strong prerequisite here is that you’ve given your original model an explicit table name in the model’s Meta options[2] that identifies the table name already being used.
As for these migrations, we do need to create migrations and to “run” them, we just don’t want the SQL executed (see the strategy section below about _why _we “need to”). So after creating the migrations using the “makemigrations” command, all we need to do is swap out the classes used to create each Migration.
To figure out how, we turn to the Django Migration class itself.
When a migration is executed, the Migration.apply method is called. This method in turns assembles a sequence of _Operations _and runs the database_forwards (or database_backwards) method for each.
Here’s what this looks like for the CreateModel class.
class CreateModel(ModelOperation): """Create a model's table.""" ... def database\_forwards(self, app\_label, schema\_editor, from\_state, to\_state): model = to\_state.apps.get\_model(app\_label, self.name) if self.allow\_migrate\_model(schema\_editor.connection.alias, model): schema\_editor.create\_model(model) def database\_backwards(self, app\_label, schema\_editor, from\_state, to\_state): model = from\_state.apps.get\_model(app\_label, self.name) if self.allow\_migrate\_model(schema\_editor.connection.alias, model): schema\_editor.delete\_model(model)
It’s pretty simple. Each method is 3 lines and just calls the method for the database backend’s schema editor to create or delete a model - in the database.
The other operations, from deleting models to adding or altering fields, all follow the same pattern.
The solution, then, is simple hack courtesy of the MRO[3].
Here’s the mixin class for creating neutered operations.
class NeuteredOperation: """ Migrations operation that does not modify the database """ def database\_forwards(self, \<...args truncated\>): pass def database\_backwards(self, \<...args truncated\>): pass
Ta-da.
I told you it was simple. Any operation using this class as a mixin (depending on order!) will have “neutered” database operation methods.
To finalize this, we need to create modified Operation classes by using the mixin. These can all be kept in one module with a copy or imported copy of the Migration class itself so that the only thing you need to swap out in your migration is the import, and everything falls into place.
**import** neutered\_migrations **as** migrations
If I’ve left out any code here that’s because it’s included in a GitHub Gist[4].
What’s all the fuss about and why bother with this migration swapping?
The goal is to refactor the application to make it easier to maintain, easier to test, easier to extend. To do that we need to break apart and move code. And on the other hand, you have production data for which we want to remove any risk of unnecessary modification just to satisfy the framework.
One of the presumptions here is that you have automated testing and deployment processes. This could involve test data, it could involve multiple team members, scripted processes over which you have little control, or processes you don’t want to disrupt. In any case, the goal is to avoid a one-off deployment just for moving some source code.
Consistent deployment and consistent, repeatable processes.
The alternatives are flaky: remembering to run ‘fake’ migrations, writing one off scripts for deployment - just say no. By updating the migrations as-is you can give the code to anyone or any service you have running and know that they can execute the same steps they always do and get the right answer.
If you have historical data migrations in the older app you should consider moving those to the new app so that you can squash all of your migrations and remove any trace of the moved models in the original app.
This may also be a good time to use the models’ Meta db_table option to rename the table to match the new app or model class name, if you’ve renamed the class. The reasons _for _are naming consistency and that this provides clear intent to anyone querying the database directly. Your migration will execute a simple query to rename the table name. I don’t have any stats to show you, but this should be a very quick query to execute.
The reasons _against _renaming the model’s table name have to do with existing table references outside of Django. If you have any outside queries or database functions or triggers that reference the table name, these have to be updated. This may or may not be worth the trouble, depending on how relevant the current table name is and the number of name references that need to be changed, and how responsibility for these entities is distributed in your organization.
Stationarily yours,
Ben
P.s. I’m planning on releasing a very small library including these “neutered” migrations that you can use with a simple “pip install” and I’ll let you know when I do.
P.p.s. I’m prettyu sure the terminology I’ve chosen here is going to rub some folks the wrong way, so if you have any suggestions for alternative terminology for these migrations shoot me an email and let me know. I’ll be waiting for your replies.
[0] Moving the Monolith (part 2): https://www.getdrip.com/broadcasts/392396119/44507ff37536f21d34783
[1] Migration with ‘–fake’ https://docs.djangoproject.com/en/2.0/ref/django-admin/#cmdoption-migrate-fake
[2] Meta Options: db_table https://docs.djangoproject.com/en/2.0/ref/models/options/#db-table
[3] New style classes model resolution order https://www.python.org/download/releases/2.3/mro/, see also Raymond Hettinger’s PyCon talk, “Super considered super!”
[4] https://gist.github.com/bennylope/07f0860aeb3ca2eb66656cfdf2396854
Learn from more articles like this how to make the most out of your existing Django site.
