This is somewhat of a shallow-level question. However, I perplexed by this trio of services.
I understand that KPL produces fast data and KCL consumes fast data produced by Kinesis. However, what I fail to understand is the if KPL and KCL make up this pair, what do we need AWS Kinesis for?
Another way to look at it: If AWS Kinesis can produce the fast data and KCL can consume it, then what we need KPL for?
Any clarifying answer is greatly appreciated.
The Kinesis Producer Library (KPL) aggregates small user-formatted records into larger records up to 1 MB to make better use of Amazon Kinesis Data Streams throughput.
While the KCL for Java supports deaggregating these records.
Refer this for more: https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html
One problem, the KCL and KPL are heavily focused on Java, but most of the data scientists love Python. One can always create amazon-kinesis-client-python library on top of Java MultiLangDaemon for interprocess communication, but it is not recommended.
AWS Kinesis is a very broad platform. Roughly, you can think of AWS Kinesis as: Kinesis Data Streams + Kinesis Video Streams + Kinesis Firehose + Kinesis Analytics. (Each one has its own purpose).
More detail here:
https://aws.amazon.com/kinesis/
Now, lets take Kinesis Data Streams , for example: What if you are a developer and you need feed data to a specific Kinesis Data Stream programatically (ie SDK)? This is where KPL comes into play. You use KPL to feed data to THAT stream.
Similar Story with KCL:
If you are a developer and you want get data ("consume") from that DATA STREAM, you use KCL.
In short: AWS Kinesis is huge platform, where KCL and KPL serve specific purposes.
TL;DR :
Details :
There's obviously more to it, but they appear to go well together if you happen to use Java. And, if you don't use KPL / KCL , you'll probably want to implement something that looks like it.
Based on my research, it looks like you have to use Java if you want to use KPL and you can use other languages with the KCL, but it looks complicated and you may have to give up some of the features that drew you to the KPL/KCL in the first place (like aggregation).
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.