Consuming DynamoDB Streams with AWS Kinesis Data Analytics

Question

I am using Kinesis Firehose to consume Dyanamo DB streams through lambda and pushing those records to S3 bucket, Glue job is running every hour to pick the records from S3 , perform deduplication and then finally insert the records to Redshift.

Is there any way I can consume the records from Dynamo Streams to 'Kinesis Data Analytics' and then perform deduplication here and insert the records in Redshift?

I have gone through some links https://issues.apache.org/jira/browse/FLINK-4582 , Consume DynamoDB streams in Apache Flink .

• Here it is mentioned that we can use FlinkKinesisConsumer to consume DynamoDB streams

. So Can we use this FlinkKinesisConsumer in Kinesis Data Analytics and then consume the Dynamo Stream directly?

Answer 1

While using Flink as Runtime for Kinesis Data Analytics.

sources : https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-sources.html

'FlinkKinesisConsumer' can be used to adapt the Dynamo DB Streams ( https://issues.apache.org/jira/browse/FLINK-4582 ).

destinations: https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-sinks.html

'FlinkKinesisFirehoseProducer ' can be used to write into 'Kinesis data firehose'. There is no direct integration with Redshift.