Pipe DynamoDB streams to a webhook

Question

I'm looking out for a solution to pipe DynamoDB Streams to my own webhook or on-prem data pipeline to process the data. Based on my exploration and also this thread, it looks like we need to connect DynamoDB streams to either

1. Kinesis or Lambda

I can't figure out another adaptor or to be more specific a generic webhook adaptor.

Is it possible to pipe the stream outside by directly hooking to my webhook?

Answer 1

Here is how I did it.

DynamoDB --> Streams (Old and New Image) --> Lambda --> (http/https or any other endpoint)

Have your DynamoDB Table create streams with old and new image of the changed data and send it to your lambda. Once the lambda is triggered it will deliver the packet for you.

You can also configure one or more batch data delivered to your lambda.

DynamoDB Streams:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

NodeJS Lambda DynamoDB Events Trigger Implementation:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html

Hope it helps.

Answer 2

2 part series on stream processing from Dynamodb to Elasticsearch:

https://medium.com/realtime-data-streaming/data-streaming-from-dynamodb-to-elasticsearch-eb2381446f43

https://medium.com/realtime-data-streaming/data-streaming-from-dynamodb-scaling-it-up-8273d23295c

The first post has some evaluation on Lambda vs KCL and both has details on the KCL approach. Hope this helps.