如果过去 1 小时内没有推送记录，如何设置 Spark 流作业的警报？

Question

I have a spark streaming job that reads and processes data from the solace queue. I want to set an alert on it if no data is consumed in last one hour. Currently, I have set a batch window as 1 minute. How can add an alert if no data is consumed continuously for an hour so that source can be notified?

Answer 1

You can keep track of it by saving the timestamp of the last received record in a hdfs file. And then in while processing micro-batch, if rdd is empty and the difference in current timestamp and timestamp in hdfs is more than an hour you can send a mail using your mailing service. If you receive some records in your micro batch you can update the timestamp in hdfs file accordingly. Your code will look something like below where you need to implement getTimeStampFromHDFS() which will return timestamp in your hdfs file and updateTimestampHDFS(currentTimestamp) in which you will update the timestamp when you received record in your micro batch.

dstream.foreachRDD{rdd => 
    if(rdd.isEmpty) {
        if((System.currentTimeMillis - getTimeStampFromHDFS()) / (1000 * 60 * 60) >= 1) sendMailAlert()
    }
    else {
        updateTimestampHDFS(System.currentTimeMillis)
    }
}

Answer 2

You have several options to do that:

• Add your implementation of a StreamListener ( API ) to the stream you are subscribed to. Then, override the onBatchCompleted to access the BatchInfo that will give you the batch time and size. With that info, you can track what happens and raise an alarm if no data has been received during a specific period.

• You also can use the REST api provided for monitoring, described here . You check it from the outside and raise the alarm if needed. For instance, it might be helpful to check /applications/[app-id]/streaming/batches