如果某些Kafka节点时间偏移未同步，则Spark流式传输作业会停止

Question

We have a spark streaming job that reads from Gnip API and send tweets to a Kafka cluster.

The Kafka cluster is installed using Cloudera Manager.

Sometimes, the cloudera manager shows a bad health message for some Kafka nodes. The bad health message is related to the NTP service. Some nodes suddenly be not synchronized with the NTP server.

Once this happen, the Spark streaming job stuck and a lot of jobs queued without processing for a long time.

Why the synchronization of Kafka nodes with the NTP server affect the Kafka producer in the spark streaming job?

Answer 1

Every partition has its leader and followers in Kafka brokers, by which Kafka provides its fault-tolerance. This mechanism is based on ZooKeeper, which uses NTP service.

If you use the default configuration, the leader will receive your data, and try it best to write into followers. It will not respond a success message until data are written into every follower. So your Spark application blocks.

You can also change your Kafka configuration to respond immediately when leader receives data or respond immediately when the leader has written data into disk.

You can find more in Kafka documents .