unable to upgrade airflow helm chart with custom values

Question

I have a cluster in Azure with the official airflow helm chart. Helm chart installs great no issues. When I attempted to upgrade the chart with my changes to the helm "values.yaml", only change is trying to add an existing postgres connection. I run into issues.

I am following the guide from https://airflow.apache.org/docs/helm-chart/stable/production-guide.html

I apologize I am new to airflow and kube.netes and I am learning as I go, any help would be appreciated. at the bottom is the current yaml. it is very long, i apologize.

Migrations pod logs

$ k logs airflow-run-airflow-migrations-lnb7n -n airflow

    DB: postgresql+psycopg2://airflow_user:***@hostname-vm.com:5432/airflow_test
    Performing upgrade with database postgresql+psycopg2://airflow_user:***@hostname-    vm.com:5432/airflow_test
    [2022-11-15T16:00:05.411+0000] {migration.py:204} INFO - Context impl PostgresqlImpl.
    [2022-11-15T16:00:05.411+0000] {migration.py:211} INFO - Will assume transactional DDL.
    [2022-11-15T16:00:05.467+0000] {db.py:1531} INFO - Creating tables
    INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
    INFO  [alembic.runtime.migration] Will assume transactional DDL.
    Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 233, in _catch_revision_errors
    yield
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 443, in _upgrade_revs
    for script in reversed(list(revs))
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 804, in iterate_revisions
    assert_relative_length=assert_relative_length,
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 1454, in _collect_upgrade_revisions
    current_revisions = self.get_revisions(lower)
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 527, in get_revisions
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 527, in <listcomp>
    return sum([self.get_revisions(id_elem) for id_elem in id_], ())
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 554, in get_revisions
    for rev_id in resolved_id
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 554, in <genexpr>
    for rev_id in resolved_id
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/revision.py", line 635, in _revision_for_ident
    resolved_id,
    alembic.script.revision.ResolutionError: No such revision or branch 'b0d31815b5a6'

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
    File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52,   in command
    return func(*args, **kwargs)
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
    return f(*args, **kwargs)
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 84, in upgradedb
    db.upgradedb(to_revision=to_revision, from_revision=from_revision,   show_sql_only=args.show_sql_only)
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 1532, in   upgradedb
    command.upgrade(config, revision=to_revision or 'heads')
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/command.py", line 322, in  upgrade
    script.run_env()
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 569, in   run_env
    util.load_python_file(self.dir, "env.py")
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 94, in   load_python_file
    module = load_module_py(module_id, path)
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 110, in   load_module_py
    spec.loader.exec_module(module)  # type: ignore
    File "<frozen importlib._bootstrap_external>", line 728, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/env.py", line 112,  in <module>
    run_migrations_online()
    File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/env.py", line 106,       in run_migrations_online
    context.run_migrations()
    File "<string>", line 8, in run_migrations
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/environment.py", line   853, in run_migrations
    self.get_context().run_migrations(**kw)
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/command.py", line 311, in upgrade
    return script._upgrade_revs(revision, rev)
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 443, in _upgrade_revs
    for script in reversed(list(revs))
    File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
    File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 265, in  _catch_revision_errors
    raise util.CommandError(resolution) from re
    alembic.util.exc.CommandError: Can't locate revision identified by 'b0d31815b5a6'

Error from kubectl after helm upgrade timeout:

$ helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug
    warning: Upgrade "airflow" failed: post-upgrade hooks failed: job failed: BackoffLimitExceeded
    Error: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded
    helm.go:84: [debug] post-upgrade hooks failed: job failed: BackoffLimitExceeded
    UPGRADE FAILED
    main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:201
    github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.5.0/command.go:872
    github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.5.0/command.go:990
    github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.5.0/command.go:918
    main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
    runtime.main
        runtime/proc.go:250
    runtime.goexit
        runtime/asm_arm64.s:1172

yaml changes -- Line 347, lines 319-321 for configmap - can post that too if need be.

I have tried a config map, and just editing the fields inside the values.yaml from helm. I tried postgres-psycopg2 and just postgres as the protocol. For host name i tried name of vm-hostname.com as well as the ip of the vm. postgres sql is set to false. I also created a new cluster with more power with no luck.

I did find the below link but none of this worked for me.

Airflow - Why is the externalDatabase configuration breaking helm upgrade?

I did create just a basic web/scheduler containers which can connect to the postrges db. I had to upgrade the db for those to connect. I also reset the db before trying to upgrade the helm chart.

    metadataConnection: 
        user: airflow_user
        pass: password
        protocol: postgresql+psycopg2
        host: vm-hostname.com
        port: 5432
        db: airflow_test
        sslmode: disable

Answer 1

I solved this by changing the airflow version in values.yaml

https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html

File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 265, in  _catch_revision_errors
raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by 'b0d31815b5a6'

revision error was referencing airflow version 2.4.2; I was attempting to use 2.4.0. Dumb mistake. hope this helps someone.