通过cli查询aws kinesis put-record

Question

This is regarding aws kinesis put-record command through AWS CLI.

I am able to input text data using kinesis cli.

aws kinesis put-record --cli-binary-format raw-in-base64-out --stream-name NagaTZeusTestStream --partition-key 1 --data 2 --region  us-west-2

Here the data is 2

But how can i put a csv file in place of 2 as data.

And how can i put a csv file which is in s3

For example:

aws kinesis put-record --cli-binary-format raw-in-base64-out --stream-name NagaTZeusStream --partition-key 1 --data s3://cona-sample-salesforce-data/testdata/ --region  us-west

In this case the file csv file in the s3 bucket should be uploaded as a data record, but kinesis is considering the s3 path itself as a data string.

Any help would be apprecieated. Thanks in advance

Answer 1

Kinesis Streams lets you write opaque blobs of data. The Kinesis PutRecord API (which is what the AWS CLI kinesis put-record command calls) expects you to give it a blob of data. If the data is stored in S3, it is your responsibility to load that data to send to Kinesis.

A common Kinesis pattern when working with "large" data is to put actual data into some other storage system (S3 being a great example) and then writing the "location" of that data (in this case, an S3 path) to Kinesis. With Kinesis Streams, your throughput (and costs) to/from Kinesis are directly affected by the amount of data you read/write. This of course requires coordination between the publisher and consumer, on the exact format (and semantics) of messages. If this is the case, you should look at what your consumers are expecting the format of your message to be.

But the moral of the story here is that Kinesis (and the CLI's put-record ) are going to put/write exactly what you give it.