Python flask-migrate data migration not saved

Question

When I add a new column and migrate data to it from a pre existing one, the column is added but the data is not migrated (only null values saved in column).

from alembic import op  # NOQA
import sqlalchemy as sa  # NOQA
from pycoin.encoding import a2b_hashed_base58  # NOQA
from binascii import hexlify  # NOQA
from datetime import datetime, timedelta  # NOQA
from sqlalchemy.ext.declarative import declarative_base  # NOQA


Base = declarative_base()
Session = sa.orm.sessionmaker()


class Farmer(Base):
    __tablename__ = 'farmer'
    id = sa.Column(sa.Integer, primary_key=True)
    btc_addr = sa.Column(sa.String(35), unique=True)  # TODO migrate to nodeid


def upgrade():
    bind = op.get_bind()
    session = Session(bind=bind)
    op.add_column('farmer', sa.Column('nodeid', sa.String(length=40),
                                      nullable=True))
    for farmer in session.query(Farmer):
        nodeid = hexlify(a2b_hashed_base58(farmer.btc_addr)[1:])
        if isinstance(nodeid, bytes):
            nodeid = nodeid.decode("utf-8")
        farmer.nodeid = nodeid
        print("saved nodeid: {0}".format(nodeid))  # called
    session.commit()

Edit:

The question has been correctly answered and I created an isolated example here to demonstrate how to do a data migration, because I could not find a good one anywhere.

Answer 1

As a general rule, it is a bad idea to use the ORM inside migration scripts. As you have already experienced, you have to do weird things to avoid errors. Clearly you've seen that and as a result decided to include a copy of your model in the script itself.

The problem with your solution is that your copied Farmer model does not have the nodeid property you are migrating to. If you include this property in the copy of the model you have in the migration script I think you will be able to perform the data migration.

But in any case, I think having models explicitly copied in the migrations can lead to errors. My recommendation is that you don't use the ORM to migrate your data. You can still have something very decent if you use the lower level SQL generation support in SQLAlchemy. You would create a temporary Table instance that represents the state of your database table at the exact point in which it needs to be migrated, then you can just issue the SQL classes in SQLAlchemy to make the changes you need.

For an example migration script structure that includes data migrations, see this example in the official Alembic documentation: http://alembic.readthedocs.org/en/latest/cookbook.html#conditional-migration-elements .

For an example that implements a data migration script similar to yours, see https://julo.ch/blog/migrating-content-with-alembic/ .