Trigger a dag in Amazon Managed Workflows for Apache Airflow (MWAA) as a part CI/CD

Question

Wondering if there is any way (blueprint) to trigger an airflow dag in MWAA on the merge of a pull request (preferably via github actions)? Thanks!

Answer 1

(Answering for Airflow without specific context to MWAA)

Airflow offers rest API which has trigger dag end point so in theory you can configure GitHub action that will run after merge of PR and trigger a dag run via REST call. In theory this should work.

In practice this will not work as you expect.

Airflow is not synchronous with your merges (even if merged dump code in the dag folder and there is no additional wait time for GitSync). Airflow has a DAG File Processing service that scans the Dag folder and lookup for changes in files. It process the changes and then a dag is registered to the database. Only after that Airflow can use the new code. This seralization process is important it makes sure different parts of airflow (webserver etc..) don't have access to your dag folder.

This means that if you invoke dagrun right after merge you are risking that it will execute an older version of your code.

I don't know what why you need such mechanism it's not very typical requirement but I'd advise you to not trying to force this idea into your deployment.

To clarify : If under a specific deployment you can confirm that the code you deployed is parsed and register as dag in the database then there is no risk in doing what you are after. This is probably a very rare and unique case.

Answer 2

You need to create a role in AWS:

1. set permission with policy airflow:CreateCliToken

    { "Action": "airflow:CreateCliToken", "Effect": "Allow", "Resource": "*" }

2. Add trusted relationship (with your account and repo)

    { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::{account_id}:oidc-provider/token.actions.githubusercontent.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "token.actions.githubusercontent.com:sub": "repo:{repo-name}:*" } } } ] }

3. In github action you need to set AWS credential with role-to-assume and permission to job

permissions: id-token: write contents: read - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v1 with: role-to-assume: arn:aws:iam::{ account_id }:role/{role-name} aws-region: {region}

4. Call MWAA using the CLI see aws ref about how to create token and run dag.