[英]Samza 0.14.1 not correctly handling OffsetOutOfRangeException exception?
We are facing an identical problem as described in this thread . 我们正面临与此线程中描述的相同问题。
Here - Samza is requesting for an Kafka partition offset that is too old (ie Kafka log has moved ahead). 在这里-Samza请求的Kafka分区偏移量太旧(即Kafka日志已向前移动)。 We are setting the property consumer.auto.offset.reset
to smallest
and therefore expecting that Samza will reset its checkpoint to earliest available partition offset in such a scenario. 我们将属性consumer.auto.offset.reset
设置为smallest
,因此,在这种情况下,Samza希望将其检查点重置为最早的可用分区偏移量。 But that is not happening we are getting exceptions of this form continually: 但这并没有发生,我们不断地收到这种形式的例外情况:
INFO [2018-08-21 19:26:20,924] [U:669,F:454,T:1,123,M:2,658]
kafka.producer.SyncProducer:[Logging_class:info:66] - [main] -
Disconnecting from vrni-platform-release:9092
INFO [2018-08-21 19:26:20,924] [U:669,F:454,T:1,123,M:2,658]
system.kafka.GetOffset:[Logging_class:info:63] - [main] - Validating offset
56443499 for topic and partition Topic3-0
WARN [2018-08-21 19:26:20,925] [U:669,F:454,T:1,123,M:2,658]
system.kafka.KafkaSystemConsumer:[Logging_class:warn:74] - [main] - While
refreshing brokers for Topic3-0:
org.apache.kafka.common.errors.OffsetOutOfRangeException: The requested
offset is not within the range of offsets maintained by the server..
Retrying
Version Details 版本详细信息
Browsing through the code, it appears that GetOffset::isValidOffset
should be able to catch the exception OffsetOutOfRangeException
and convert it to a false value. 浏览代码,似乎GetOffset::isValidOffset
应该能够捕获异常OffsetOutOfRangeException
并将其转换为假值。 But it appears that this not happening. 但是看来这没有发生。 Could there be a mismatch in package
of the Exception
? Exception
package
中是否可能存在不匹配? GetOffSet class is catching the exception import kafka.common.OffsetOutOfRangeException
, but from logs, it appears that the package of this class is different. GetOffSet类正在捕获异常import kafka.common.OffsetOutOfRangeException
,但是从日志来看,该类的软件包似乎有所不同。 Could this be the reason? 这可能是原因吗?
def isValidOffset(consumer: DefaultFetchSimpleConsumer, topicAndPartition: TopicAndPartition, offset: String) = {
info("Validating offset %s for topic and partition %s" format (offset, topicAndPartition))
try {
val messages = consumer.defaultFetch((topicAndPartition, offset.toLong))
if (messages.hasError) {
KafkaUtil.maybeThrowException(messages.error(topicAndPartition.topic, topicAndPartition.partition).exception())
}
info("Able to successfully read from offset %s for topic and partition %s. Using it to instantiate consumer." format (offset, topicAndPartition))
true
} catch {
case e: OffsetOutOfRangeException => false
}
}
Also, it Appears that BrokerProxy class - the caller of GetOffset
would print a log "It appears that..."
in case it gets a false value, but it is not logging this line (indicating that some Exception generated in GetOffset
method is going uncaught and being propagated up): 此外,似乎BrokerProxy类-调用者GetOffset
将打印日志"It appears that..."
的情况下,它得到一个错误的值,但它不记录此行(表明在产生了一些异常GetOffset
方法是怎么回事未被捕获并向上传播):
def addTopicPartition(tp: TopicAndPartition, nextOffset: Option[String]) = {
debug("Adding new topic and partition %s to queue for %s" format (tp, host))
if (nextOffsets.asJava.containsKey(tp)) {
toss("Already consuming TopicPartition %s" format tp)
}
val offset = if (nextOffset.isDefined && offsetGetter.isValidOffset(simpleConsumer, tp, nextOffset.get)) {
nextOffset
.get
.toLong
} else {
warn("It appears that we received an invalid or empty offset %s for %s. Attempting to use Kafka's auto.offset.reset setting. This can result in data loss if processing continues." format (nextOffset, tp))
offsetGetter.getResetOffset(simpleConsumer, tp)
}
debug("Got offset %s for new topic and partition %s." format (offset, tp))
nextOffsets += tp -> offset
metrics.topicPartitions.get((host, port)).set(nextOffsets.size)
}
Could this be due to mismatch in Kafka client library version that we are using? 这可能是由于我们使用的Kafka客户端库版本不匹配造成的吗? Is there a recommended Kafka client version we should use with Samza 0.14.1 (assuming that Kafka server is 1.x)? Samza 0.14.1是否应该使用推荐的Kafka客户端版本(假设Kafka服务器为1.x)?
Any help regarding this will be greatly appreciated. 任何与此有关的帮助将不胜感激。
The above is a bug in samza 0.14.0 and 0.14.1. 上面是samza 0.14.0和0.14.1中的错误。 SAMZA-1822 is the bug id. SAMZA-1822是错误ID。
This is also discussed in the samza mailing list . samza 邮件列表中也对此进行了讨论。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.