How can we use streaming in spark from multiple source? e.g First take data from HDFS and then consume streaming from Kafka

Question

The problem arise when I already have a system and I want to implement a Spark Streaming on top. I have 50 million rows transactional data on MySQL, I want to do reporting on those data. I thought to dump the data into HDFS. Now, Data are coming everyday also in DB and I am adding KAFKA for new data.

I want to know how can I combine multiple source data and do analytics in real-time (1-2 minutes delay is ok) and save those results because future data needs previous results.

Answer 1

Joins are possible in SparkSQL, but what happens when you need to update data in mysql? Then your HDFS data becomes invalid very quickly (faster than a few minutes, for sure). Tip: Spark can use JDBC rather than need HDFS exports.

Without knowing more about your systems, I say keep the mysql database running, as there is probably something else actively using it. If you want to use Kafka, then that's a continous feed of data, but HDFS/MySQL are not. Combining remote batch lookups with streams will be slow (could be more than few minutes).

However, if you use Debezium to get data into Kafka from mysql , then you then have data centralized in one location, and then ingest from Kafka into an indexable location such as Druid, Apache Pinot, Clickhouse, or maybe ksqlDB to ingest.

Query from those, as they are purpose built for that use case, and you don't need Spark. Pick one or more as they each support different use cases / query patterns.