在 Helm (Kube.netes) 中使用 Values.yaml 设置 airflow 连接

Question

Airflow Version - 2.3.0 Helm Chart - Apache-airflow/airflow

I have been working on setting up airflow using helm on kube.netes. Currently, I am planning to set airflow connections using the values.yaml file and env variables instead of configuring them up on the webUI.

I believe the settings to tweak, to set the connections, are:

extraSecrets: {}
# eg:
# extraSecrets:
#   '{{ .Release.Name }}-airflow-connections':
#     type: 'Opaque'
#     data: |
#       AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string'
#       AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string'
#     stringData: |
#       AIRFLOW_CONN_OTHER: 'other_conn'
#   '{{ .Release.Name }}-other-secret-name-suffix':
#     data: |
#        ...

I am not sure how to set all the key-value pairs for a databricks/emr connection, and how to use the kube.netes secrets (already set up as env vars in pods) to get the values

#extraSecrets:
#  '{{ .Release.Name }}-airflow-connections':
#    type: 'Opaque'
#    data:
#      AIRFLOW_CONN_DATABRICKS_DEFAULT_two:
#         conn_type: "emr"
#         host: <host_url>
#         extra:
#          token: <token string>
#          host: <host_url>

It would be great to get some insights on how to resolve this issue

I looked up this link: managing_connection on airflow

Tried Changes in values.yaml file:

#extraSecrets:
#  '{{ .Release.Name }}-airflow-connections':
#    type: 'Opaque'
#    data:
#      AIRFLOW_CONN_DATABRICKS_DEFAULT_two:
#         conn_type: "emr"
#         host: <host_url>
#         extra:
#          token: <token string>
#          host: <host_url>

Error Occurred: While updating helm release: extraSecrets.{{.Release.Name }}-airflow-connections expects string, got object

Answer 1

Airflow connections can be set using Kube.netes secrets and env variables.

For setting secrets, directly from the cli, the easiest way is to

• Create a kube.netes secret

  • The secret value (connection string) has to be in the URI format suggested by airflow

    my-conn-type://login:password@host:port/schema?param1=val1&param2=val2

• Create an env variable in the airflow-suggested-format

  • Airflow format for connection - AIRFLOW_CONN_{connection_name in all CAPS}

• set the value of the connection env variable using the secret

How to manage airflow connections: here

Example, To set the default databricks connection (databricks_default)in airflow -

• create secret

    kubectl create secret generic airflow-connection-databricks \
        --from-literal=AIRFLOW_CONN_DATABRICKS_DEFAULT='databricks://@<DATABRICKS_HOST>?token=<DATABRICKS_TOKEN>'

• In helm's (values.yaml), add new env variable using the secret:

    envName: "AIRFLOW_CONN_DATABRICKS_DEFAULT"
                secretName: "airflow-connection-databricks"
                secretKey: "AIRFLOW_CONN_DATABRICKS_DEFAULT"

Some useful links:

• Managing Airflow connections
• Databricks databricks connection