如何在airflow dag中用pandas直接从s3读取excel文件？

Question

I am trying to read an excel file from s3 inside an aiflow dag with python, but it does not seem to work. It is very weird because it works when I read it from outside airflow with pd.read_excel(s3_excel_path).

What I did:

• Set AWS credential in airflow (this works well as I can list my s3 bucket)
• Install pandas, s3fs in my Docker environment where I run Airflow
• Try to read the file with pd.read_excel(s3_excel_path)

As I said, it works when I try it outside of Airflow. Moreover, I don't get any error, the dag just continues to run undefinitely (at the step where it is supposed to read the file) and nothing happens, even if I wait 20 minutes.

(I would like to avoir to download the file from s3, process it and then upload it back to s3, which is why I am trying to read it directly from s3)

Note: I does not work with csv as well.

EDIT: Likewise, I can't save my dataframe directly to S3 with df.to_csv('s3_path') in airflow dag while I can do it in python

Answer 1

To read data files stored in S3 using pandas, you have two options, download them using boto3 (or AWS CLI) and read local files, which is the solution you are not locking for, and use s3fs API supported by pandas:

import os

import pandas as pd

AWS_S3_BUCKET = os.getenv("AWS_S3_BUCKET")
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_SESSION_TOKEN = os.getenv("AWS_SESSION_TOKEN")

key = "path/to/excel/file"

books_df = pd.read_excel(
    f"s3://{AWS_S3_BUCKET}/{key}",
    storage_options={
        "key": AWS_ACCESS_KEY_ID,
        "secret": AWS_SECRET_ACCESS_KEY,
        "token": AWS_SESSION_TOKEN,
    },
)

to use this solution, you need to install s3fs and apache-airflow-providers-amazon

pip install s3fs
pip install apache-airflow-providers-amazon