使用Storm螺栓或Spark流与MongoDB丰富数据

Question

I want to create a Storm Spout that reads data from a topic of Apache Kafka and sends this data to a Storm bolt that connects to MongoDB and query the message I collected from Kafka to enrich the data. For example: I have a personID (that I got through a message from Kafka) and I want to query the person address in MongoDB, using this personID. In my MongoDB collection every document has personID and the address.

Can anyone give me an example of that, please? An example using Spark-streaming would also be really great.

Answer 1

I would approach this thus:

1. Stream all your data into Kafka, including your MongoDB enrichment source (addresses, etc). You can use Kafka Connect (part of Apache Kafka) to do this. Check out this article: Streaming Data from MongoDB into Kafka with Kafka Connect and Debezium .

2. Perform your data enrichment using Kafka Streams , or KSQL . Kafka Streams is part of Apache Kafka, and is a Java API. KSQL runs on top of Kafka Streams, and gives you a SQL interface to declare your stream transformations. You can see an example, including joins, in this article .

   1. KSQL introduction
   2. Kafka Streams introduction

3. Optionally, if you want to store the resulting enriched data elsewhere, use Kafka Connect to stream it from the Kafka topic to the target.