Airflow Operational error sqlalquemy (sqlite3.OperationalError)

Question

I'm trying to install airflow, and I get this error when I try to create the db or run standalone. I tried to change the version of sqlalchemy but it didn't work.

os: manjaro | sqlalchemy version: 1.4 | airflow version: 2.3.0

standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, Add 
sqlite3.OperationalError: duplicate column name: operator

The above exception was the direct cause of the following exception:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: operator
[SQL: ALTER TABLE task_instance ADD COLUMN operator VARCHAR(1000)]
(Background on this error at: http://sqlalche.me/e/14/e3q8)

Something to fix it?

Answer 1

what I found I checked out the code files pointed out by 'Shi Chen' two files are responsible for this behavior.

33ae817a1ff4_add_kubernetes_resource_checkpointing.py
86770d1215c0_add_kubernetes_scheduler_uniqueness.py

Both the files are migration files using alembic and sqlalchemy libraries I found that following sqlalchemy code written in file 33ae817a1ff4_add_kubernetes_resource_checkpointing.py

def upgrade():

    columns_and_constraints = [
        sa.Column("one_row_id", sa.Boolean, server_default=sa.true(), primary_key=True),
        sa.Column("resource_version", sa.String(255))
    ]

    conn = op.get_bind()

    # alembic creates an invalid SQL for mssql dialect
    if conn.dialect.name not in ('mssql'):
        columns_and_constraints.append(sa.CheckConstraint("one_row_id", name="kube_resource_version_one_row_id"))

    table = op.create_table(
        RESOURCE_TABLE,
        *columns_and_constraints
    )

    op.bulk_insert(table, [
        {"resource_version": ""}
    ])

is interpreted into following SQL query which is not correct

CREATE TABLE  kube_resource_version (one_row_id BOOL NOT NULL DEFAULT
true,  resource_version VARCHAR(255), PRIMARY KEY (one_row_id), 
CONSTRAINT kube_resource_version_one_row_id CHECK (one_row_id),  CHECK(one_row_id IN (0, 1))

Instead the SQL query should be some what like this

CREATE TABLE  kube_resource_version (one_row_id BOOL NOT NULL DEFAULT
true,  resource_version VARCHAR(255), PRIMARY KEY (one_row_id), 
CONSTRAINT kube_resource_version_one_row_id CHECK (one_row_id IN (0, 1)))

The link provided by 'skadya' was helpful I got the system to work after making the changes in the code of the two above mentioned files.

you simple need to change the following code from

if conn.dialect.name not in ('mssql'):
columns_and_constraints.append(
sa.CheckConstraint("one_row_id", name="kube_resource_version_one_row_id"))

to

if conn.dialect.name not in ('mssql', 'mysql'):
columns_and_constraints.append(
sa.CheckConstraint("one_row_id", name="kube_resource_version_one_row_id")