Airflow connection for a single DAG

Question

I am creating a connection with a Google Service Account in my Google Cloud Composer that privilegies a DAG for a specific use case with deals with sensitive data, the point is that I want that connection to be exclusive for a certain DAG and no other could see or use it.

Is there a way of doing it?

Answer 1

Currently this is not possible in airflow, and even you cannot implement that using a custom backend secret or another solution, where the connection is not a context variable, and it's accessible from anywhere in airflow not only from a run context.

Answer 2

Infortunately the service account given to Cloud Composer in the creation of cluster, is for all DAGs of this cluster.

It can be too much, but maybe you can create another Cloud Composer cluster 2 (GKE autopilot), with the minimum sizing for machines, containing this DAG that treats sensitive data. Then you can give a SA with the needed privileges to this cluster.

The disadvantage of this solution is you will have a higher cost, because you have a second cluster. It will increases the cost even if the machine sizes are low. It is worth noting that Composer 2 with GKE autopilot is cheaper that classical GKE cluster.

Maybe another solution, if the rework is not too important, you can rewrite only your DAG treating sensitive data to Cloud Workflow. Cloud Workflow is serverless and you can give it a dedicated service account.