Trying to connect to aws s3 with boto3 by passing aws credentials as variables to airflow macros

Question

I have one airflow connection that looks like this:

• Conn id: my_conn_id
• Conn type: s3
• Host: my_host
• Login: abcd

I tried to connect to my s3 using boto3 with the following code without luck

s3client = boto3.client(
        service_name='s3',
        region_name='us-east-1',
        aws_access_key_id="{{ conn.conn_id.host }}",
        aws_secret_access_key="{{ conn.conn_id.host }}"
    )

Any help would be appreciated

Answer 1

Your connection is misconfigured. Enter your aws_access_key_id as login and aws_secret_key_id as password. When you generated your AWS keys, you got both the access key and secret key - you will need them both. If you don't have the secret key, regenerate the keys in AWS. See connection details here - Airflow AWS connection

In Airflow, you should use the S3Hook to generate a boto3 S3 client if you need to, but check out the functionality of the S3Hook first to see if you can use it to do your task. See functions here - S3Hook source code

from airflow.providers.amazon.aws.hooks.s3 import S3Hook
s3client = S3Hook(aws_conn_id=my_conn_id).get_conn()