How to read data from s3 bucket to Kinesis Stream

Question

I am new to Kinesis. I want to read data from s3 bucket to Kinesis stream. I have created a kinesis stream called 'mystream'. In my project every 3 month a record will be uploaded to the s3. At the same time I need to read the record to kinesis stream. I have created Lamda for reading the file and save it into the RDS DB instance. I found in lambda project there is one kinesis event. Is this the way to do the thing? please help me to get this. I don't know how to start it and all.. It would be great if some one can help me....

Answer 1

If my understanding is right your question is about how to write s3 object data into kinesis.

You can create an event Notification in S3 for lambda. In events, you can select the event (ObjectCreate (All), put, post). After Successful creation for each object create action you get notifications in lambda. In lambda event context you get the s3 object information of each invocation.

Now you can read about the object information and write it in kinesis stream.

Links for AWS S3 and Kinesis Stream using Java

https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html

https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-sdk.html

Answer 2

AWS suggests the use of AWS DMS for ingesting files from S3 to Kinsis.

https://aws.amazon.com/blogs/big-data/streaming-data-from-amazon-s3-to-amazon-kinesis-data-streams-using-aws-dms/

Answer 3

With the AWS SDK release, you may need to change the way you're reading the S3 file into your Lambda function. The contents now come back as a stream:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_ResponseSyntax

You can stream the data into your Lambda: https://carova.io/snippets/read-data-from-aws-s3-with-nodejs

Then inside your Lambda Function, push the contents into Kinesis Pushing AWS Lambda data to Kinesis Stream