在 DAG 中使用 boto3 时，Apache airflow 无法找到 AWS 凭证

Question

Running an instance of Airflow on ECS Fargate. The problem is I cannot run the code to call an existing Glue Job within the DAG. Below is the DAG script.

import boto3
import os
import logging
import time
import sys
import botocore
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from airflow.providers.amazon.aws.sensors.s3_key import S3KeySensor

default_args = {
    'owner': 'test',
    'start_date': datetime(2021, 1, 4),
    'depends_on_past': False,
    'provide_context': True
}

dag = DAG('run_glue_job',
          description='Executes Glue job.',
          schedule_interval=None,
          catchup=False,
          default_args=default_args)

def task_1(**kwargs):
    print('recieved trigger')

    glue = boto3.client('glue', 'us-east-1')
    response = glue.start_job_run(JobName='airflow-dev-job')
    print(response['JobRunId'])

    return response['JobRunId']

def task_2(**kwargs):
    print('send email message')

    return 'passed' 

t1 = PythonOperator(
    task_id = 'execute_glue_job',
    dag = dag,
    python_callable = task_2
)

t2 = PythonOperator(
    task_id = 'send_email_notification',
    dag = dag,
    python_callable = task_3
)

t1 >> t2

I am returning a credential error.

File "/usr/local/airflow/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

I tried setting the task_role_arn in the aws_default under the connections; however, I am still not able to get around the issue.

Answer 1

I was able to solve the problem by adding the correct task role in the task definition in ECS.

Make sure the task role assigned has all policies attached for services that you are trying to access/run via Airflow.