Where do I put a custom config (.cfg) file in AWS MWAA Airflow?

Question

I have a config file, dev.cfg, that looks like this:

[S3]
bucket_name = my-bucket

I need this in my code to do S3 things. I do not have the access (nor will I be given it) to modify the environmental or config variables in the AWS console. My only method of putting files in S3 is the CLI ( aws s3 cp... ).

This is the project directory structure in S3:

my-bucket/
  dags/
    dev.cfg
    some_dag_file.py
  plugins.zip
  requirements.txt

In the plugins.zip file, there is a plugin that should set the path to dev.cfg to an env var (DEV_CONFIG_PATH) that my code uses:

import os

from airflow.plugins_manager import AirflowPlugin

os.environ["DEV_CONFIG_PATH"] = os.path.join(
    os.getenv('AIRFLOW_HOME'), 'dags', 'dev.cfg')


class EnvVarPlugin(AirflowPlugin):
    name = 'env_var_plugin'

However, I'm getting an import error in the Airflow UI:

configparser.NoSectionError: No section: 'S3'

Any help is appreciated.

I'm using MWAA with Airflow version 2.4.3 running python 3.10.

Answer 1

Try asking your infra team to update the config: core.lazy_load_plugins: False