使用 Apache Airflow 编辑存储在 AWS S3 中的 CSV 无需下载

Question

I have a project that requires large amounts of CSV data to be transformed regularly. This data will be stored in S3 and I am using an EC2 instance running Ubuntu server 16.04 to perform edits to the data and Apache Airflow to route the data. Downloading and reuploading this data to S3 is quite expensive, is there a way I can edit this CSV data in memory without downloading the file to local storage on the Ubuntu instance?

Thank you in advance

Answer 1

In general you could think of program that will fetch the CSV file from s3 (using s3 sdk) and store it and transform in memory and then save back to s3. But it will still require "downloading and reuploading". The only difference is that file won't be physically stored to local disk but kept in program memory.

You could also use s3fs to mount s3 bucket to a directory on server and perform requested operations directly on the files. But they still need to be downloaded from s3 and reuploaded there (although it will be on-the-fly and invisible to you).

Hope that helps.