从 Microsoft CSV 读取大数据并写入 AWS 中的 Microsoft Excel 的策略

Question

Kindly assist in deciding the strategy to do Extraction, Transformation and Loading (ETL) work flow with Amazon AWS offerings. I am a newbie in Amazon Cloud. My use case is to read thousands of record rows from Microsoft CSV file. My intention is to write out this file into Microsoft Excel document. I want to store this excel file in single piece within S3 bucket as an object. Currently, I am doing this proof of concept with AWS Lambda. My issue is /tmp size is exceeding beyond 512 MB even if I select 6 GB RAM size in lambda specification. I do not do any disk operation. I read all of S3 csv file content once, in RAM and write out to excel, in RAM.

Answer 1

You should consider using aws glue for large volume data transformations. Lambda may be Ill suited to the task