无法将数据从AWS Kinesis存档到Glacier

Question

I am working on a Data processing application hosted as a web service on an EC2, each second a small data file (less than 10KB) in .csv format is generated.

Problem Statement: Archive all the data files generated to Amazon Glacier.

My Approach : As data files are very small. I store the files in AWS Kinesis and after few hours i flush data to S3 (because i cannot find a direct way to put data from Kinesis to Glacier) and using S3 lifecycle management at the end of the day i archive all the objects to Glacier.

My Questions :

1. Is there a way to transfer data to Glacier directly from Kinesis ?

2. Is it possible to configure Kinesis to flush data to S3/Glacier at the end of the day ? Is there any time or memory limitation upto which Kinesis can hold data ?

3. If Kinesis cannot transfer data to Glacier directly. Is there a workaround for this like - can i write a lambda function which can fetch data from Kinesis and archive it to Glacier ?

4. Is it possible to merge all the .csv file at Kinesis or S3 or Glacier level ?

5. Is Kinesis suitable for my usecase ? Is there anything else i can use ?

I would be grateful if someone can take time and answer my questions and point me to some references. Please let me know if there is a flaw in my approach or if there is a better way to do so.

Thanks.

Answer 1

1. You can't directly put data from Kinesis into Glacier (unless you want to put the 10kb filea directly into Glacier)
2. You could consider Kinesis Data Firehose as a way of flushing 15min. Increments of data to S3
3. You can definitely do that. Glacier allows direct uploads so there's no need to upload to S3 first
4. You could use Firehose to flush to S3 then transform and aggregate using Athena then transition that file to Glacier. Or you use Lambda directly and upload straight to Glacier.
5. Perhaps streaming data into Firehose would make more sense. Depending on your exact needs IoT Analytics might also be interesting.

Reading your question again, seeing you use csv files, I would highly recommend using the Kinesis > S3 > Athena > Transition to glacier approach