I have a use case where I need to save a parquet file from Lambda to S3. I need to store the events triggered in the Lambda function in append mode..
I tried with Avro but it doesn't permit to store the data in append mode in a same parquet file..
Till now, I found that just spark permit to store data in parquet file in append mode... Then, I could use Lambda to submit a spark job that store the data. What do you think about this possible solution?
However, really doesn't exists a solution without the use of spark? Thanks in advance
You can't append data to any file stored in S3, no matter the format. Sorry. It's just one of those differences between S3 and a filesystem.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.