简体   繁体   English

在 Kafka 消费者配置中将 max.poll.interval.ms 设置为大于 request.timeout.ms 的负面影响是什么

[英]What is negative effects of setting max.poll.interval.ms larger than request.timeout.ms in Kafka consumer configs

According to Kafka documentation;根据 Kafka 文档;

The new Java Consumer now supports heartbeating from a background thread.新的 Java Consumer 现在支持来自后台线程的心跳。 There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default).有一个新的配置 max.poll.interval.ms 控制消费者主动离开组之前轮询调用之间的最长时间(默认为 5 分钟)。 The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing, so we have changed its default value to just above 5 minutes.配置 request.timeout.ms 的值必须始终大于 max.poll.interval.ms 因为这是消费者重新平衡时 JoinGroup 请求可以在服务器上阻塞的最长时间,因此我们更改了其默认值到 5 分钟以上。

But I couldn't understand the possible results of making max.poll.interval.ms larger than request.timeout.ms.但我无法理解使 max.poll.interval.ms 大于 request.timeout.ms 的可能结果。

Can anybody explain this part of document:任何人都可以解释这部分文件:

because this is the maximum time that a JoinGroup request can block on the server while the consumer is rebalancing因为这是消费者重新平衡时 JoinGroup 请求可以在服务器上阻塞的最长时间

Now I'am using these consumer parameters in my project:现在我在我的项目中使用这些消费者参数:

kafka.consumer.session.timeout.ms=30000
kafka.consumer.heartbeat.interval.ms=10000
kafka.consumer.request.timeout.ms=31000
kafka.consumer.max.poll.interval.ms=259200000
kafka.consumer.max.partition.fetch.bytes=10242880

What would be the negative effects of these usage?这些用法会产生什么负面影响?

Note: My kafka-clients version is 1.1.0注意:我的 kafka-clients 版本是 1.1.0

UPDATE更新

From Kafka version 2.0.0 this rule is not valid anymore.从 Kafka 2.0.0 版开始,此规则不再有效。 From Kafka docs Notable changes in 2.0.0:来自 Kafka 文档 2.0.0 中的显着变化:

Also as part of KIP-266, the default value of request.timeout.ms has been changed to 30 seconds.同样作为 KIP-266 的一部分,request.timeout.ms 的默认值已更改为 30 秒。 The previous value was a little higher than 5 minutes to account for maximum time that a rebalance would take.先前的值略高于 5 分钟,以考虑重新平衡所需的最长时间。 Now we treat the JoinGroup request in the rebalance as a special case and use a value derived from max.poll.interval.ms for the request timeout.现在我们将重新平衡中的 JoinGroup 请求视为一种特殊情况,并使用从 max.poll.interval.ms 派生的值作为请求超时。 All other request types use the timeout defined by request.timeout.ms所有其他请求类型使用 request.timeout.ms 定义的超时

request.timeout.ms specifies how long the consumer will wait for a response from the broker. request.timeout.ms指定消费者等待代理响应的时间。

max.poll.interval.ms is used in a couple of use cases. max.poll.interval.ms用于几个用例。 It first specifies how often the consumer was to intereact with brokers (to confirm it's alive) but it's also used when the consumer joins a group.它首先指定消费者与代理交互的频率(以确认它是活着的),但在消费者加入组时也会使用它。 In this case, it indicates how long the broker can take to respond to the join group request.在这种情况下,它指示代理可以花费多长时间来响应加入组请求。

So if max.poll.interval.ms is larger than request.timeout.ms , you see that the broker may take longer to respond than the consumer will wait.因此,如果max.poll.interval.ms大于request.timeout.ms ,您会看到代理的响应时间可能比消费者等待的时间更长。 So the consumer could timeout requests.所以消费者可以超时请求。

In many cases, the defaults values for these 2 configurations are relatively good.在很多情况下,这两种配置的默认值都比较好。 Instead of overriding them to the large values I see in your question, you should first start by clearly identify how you want to change the default behaviour.不要将它们覆盖为我在您的问题中看到的大值,您应该首先清楚地确定您想要如何更改默认行为。

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Kafka 生产者配置:为什么 request.timeout.ms 应该大于 replica.lag.time.max.ms - Kafka producer config: Why request.timeout.ms should be larger than replica.lag.time.max.ms Consumer CommitFailedException - 后续调用 poll() 之间的时间长于配置的 max.poll.interval.ms - Consumer CommitFailedException - time between subsequent calls to poll() was longer than the configured max.poll.interval.ms 更改Kafka配置max.poll.interval.ms导致一名消费者卡在(重新)加入组中 - Change Kafka configuration max.poll.interval.ms causes one consumer stuck at (Re-)joining group Spring 云 stream 卡夫卡消费者卡在长时间运行的作业和 max.poll.interval.ms 的大值 - Spring cloud stream Kafka consumer stuck with long running job and large value for max.poll.interval.ms Spring Kafka,覆盖 max.poll.interval.ms? - Spring Kafka, overriding max.poll.interval.ms? Kafka poll 和 max.poll.interval.ms - 批处理最佳实践 - Kafka poll and max.poll.interval.ms - Batch Processing best practices 如何知道在 java kafka 应用程序客户端中是否达到了 max.poll.interval.ms? - How to know if max.poll.interval.ms is reached in java kafka application client? 为指定主题配置max.poll.interval.ms - Configure max.poll.interval.ms for a specified topic 当Kafka消费者不使用订阅方法时,是否检查session.timeout.ms或max.pool.interval.ms? - When Kafka consumer don't use subscribe method, do it check session.timeout.ms or max.pool.interval.ms? Apache Kafka-了解在poll()方法中作为参数存在的超时与fetch.max.wait.ms之间的关系 - Apache Kafka- Understanding the relationship between timeout present as a parameter in the poll() method and fetch.max.wait.ms
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM