[英]Samza equivalent of Kafka Consumer - Manual Offset Control (enable.auto.commit = false)
We have Samza tasks which reads messages from Kafka Output stream but if there is any retryable failure while processing the message then i would want my Samza task to read the same message again and reprocess it. 我们有Samza任务,它从Kafka Output流中读取消息,但是如果在处理消息时出现任何可重试的失败,那么我希望我的Samza任务再次读取同一消息并重新处理它。 And after successfully processing the message acknowledge it for checkpointing. 在成功处理该消息后,确认该消息用于检查点。
Is there a way to manually control the checkpoint(just like what Kafka Consumer provides "Manual Offset Control" by setting enable.auto.commit to false : https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html ) 有没有一种方法可以手动控制检查点(就像Kafka Consumer通过将enable.auto.commit设置为false来提供“手动偏移控制”一样: https ://kafka.apache.org/0100/javadoc/index.html?org /apache/kafka/clients/consumer/KafkaConsumer.html )
I came across this doc https://samza.apache.org/learn/documentation/0.13/jobs/reprocessing.html which talks about reprocessing previously processed data but it is not offering any acknowledge based checkpoint control. 我遇到了这个文档https://samza.apache.org/learn/documentation/0.13/jobs/reprocessing.html ,该文档讨论了重新处理先前处理的数据,但是它不提供任何基于确认的检查点控制。
Found relate issues https://github.com/zendesk/ruby-kafka/issues/304 找到相关问题https://github.com/zendesk/ruby-kafka/issues/304
Sid, 席德
You can perform manual commit from your StreamTask. 您可以从StreamTask执行手动提交。 If you set task.commit.ms
is set to -1, then auto-commit in the Samza job is disabled. 如果将task.commit.ms
设置为-1,则将禁用Samza作业中的自动提交。 In such a case, the task should manually trigger a commit by invoking taskCoordinator.commit()
when you are ready to acknowledge. 在这种情况下,当您准备好确认时,任务应该通过调用taskCoordinator.commit()
来手动触发提交。
You can find documentation regarding checkpointing here - http://samza.apache.org/learn/documentation/0.13/container/checkpointing.html . 您可以在此处找到有关检查点的文档-http: //samza.apache.org/learn/documentation/0.13/container/checkpointing.html 。 I believe documentation is insufficient in the website as it doesn't elaborately cover the manual commit scenario. 我认为网站上的文档不足,因为它没有详细介绍手动提交方案。 The configuration table also needs to be updated so that it is obvious to the user that manual commit is supported. 配置表也需要更新,以使用户明显支持手动提交。
HTH :) HTH :)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.