Amazon Kinesis Firehose to S3

Question

I was asked to write a code to send a .csv file to S3 using Amazon Kinesis Firehose. But as someone who has never used Kinesis, I have no idea how I should do this. Can you help with this, or if you have a code that does this job, it can also help (Java or Scala).

csv data should be sent to Kinesis Firehose to be written to a S3 bucket in gzip format using a Firehose client application.

Thanks in advance.

Answer 1

Firstly, Firehose is streaming to send a record (or records) to a destination, not a file transfer such as copy a csv file to S3. You can use S3 CLI commands if you need to copy files from somewhere to S3.

So please first make sure what you need to do is streaming or file copy. If it is not streaming, then I wonder why Firehose.

There are multiple input source you can use. First better decide which way to use.

If you use JAVA+AWS SDK, then probably PutRecord API call would be the way

• Writing to Kinesis Data Firehose Using the AWS SDK
• aws-sdk-java/src/samples/AmazonKinesisFirehose/
• Put data to Amazon Kinesis Firehose delivery stream using Spring Boot

If you can use AWS Amazon Linux to send the data to Firehose, Firehose Agent will be easier. It just monitor a file and can send the deltas to S3.

• enter link description here