如何正确使用 Blocks 在 Prefect 中加载 AWS S3 凭证？

Question

I am using Prefect. And I tried to download a file from S3.

When I hard coded the AWS credentials, the file can be downloaded successfully:

import asyncio

from prefect_aws.s3 import s3_download
from prefect_aws.credentials import AwsCredentials

from prefect import flow, get_run_logger


@flow
async def fetch_taxi_data():
    logger = get_run_logger()
    credentials = AwsCredentials(
        aws_access_key_id="xxx",
        aws_secret_access_key="xxx",
    )
    data = await s3_download(
        bucket="hongbomiao-bucket",
        key="hm-airflow/taxi.csv",
        aws_credentials=credentials,
    )
    logger.info(data)

if __name__ == "__main__":
    asyncio.run(fetch_taxi_data())

Now I tried to load the credentials from Prefect Blocks .

I created a AWS Credentials Block:

However,

aws_credentials_block = AwsCredentials.load("aws-credentials-block")
data = await s3_download(
    bucket="hongbomiao-bucket",
    key="hm-airflow/taxi.csv",
    aws_credentials=aws_credentials_block,
)

throws the error:

AttributeError: 'coroutine' object has no attribute 'get_boto3_session'

And

aws_credentials_block = AwsCredentials.load("aws-credentials-block")
credentials = AwsCredentials(
    aws_access_key_id=aws_credentials_block.aws_access_key_id,
    aws_secret_access_key=aws_credentials_block.aws_secret_access_key,
)
data = await s3_download(
    bucket="hongbomiao-bucket",
    key="hm-airflow/taxi.csv",
    aws_credentials=credentials,
)

throws the error:

AttributeError: 'coroutine' object has no attribute 'aws_access_key_id'

I didn't find any useful document about how to use it.

Am I supposed to use Blocks to load credentials? If it is, what is the correct way to use Blocks correctly in Prefect? Thanks!

Answer 1

I just found the official demo in the screenshot of the question is misleading. It misses an await .

After adding await , it works now!

aws_credentials_block = await AwsCredentials.load("aws-credentials-block")
data = await s3_download(
    bucket="hongbomiao-bucket",
    key="hm-airflow/taxi.csv",
    aws_credentials=aws_credentials_block,
)

I will open a ticket to report the issue.