使用Apache Spark / Apache Flink进行扩展

Question

I plan an application that reads from Apache Kafka and after (potentially time-consuming) processing saves data to a database.

My case are messages, not streams, but for scalability I'm thinking about plugging this into Spark or Flink, but can't grasp how these scale: should my app, when a part of Spark/Flink, read some data from Kafka and then exit or keep reading continuously?

How will then Spark/Flink decide they must spawn more instances of my app to improve throughput?

Thanks!

Answer 1

In Apache Flink you can define the parallelism of the operations by setting the env.setParallelism(#parallelism) to make all operators run with #parallelism parallel instances, or even you can define/override it per operator such as dataStream.map(...).setParallelism(#parallelism); .

For more info Check Flink docs https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/parallel.html .

Regarding reading from Kafa you can define the parallel receivers (same group) to scale up/down with the Kafka topic partitions : env.addSource(kafkaConsumer).setParallelism(#topicPartitions)

Check Kafka documentation for more info about Kafka topic and partitions and consumer group : https://kafka.apache.org/documentation/ .

Note that if you don't specify the parallelism level inside the Flink program and you deploy it on local Flink cluster. The value of parallelism.default parameter inside the configs file flinkDir/conf/flink-conf.yaml will be used. Unless you specify it by the -p like ./bin/flink run .... -p #parallelism . check Flink cli options .