![](/img/trans.png)
[英]Kafka Mirror Maker: consumer vs consumer thread number and producer number
[英]Mirror maker thread failure
當嘗試從docker堆棧中的服務運行Kafka MirrorMake時,我配置了群體網絡,我從服務日志中收到錯誤消息:
mirroring_mirror_maker | [2019-01-10 12:08:49,322] ERROR [mirrormaker-thread-1] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
mirroring_mirror_maker | java.lang.IllegalStateException: No entry found for connection 2147482646
我的consumer.properties
代碼:
bootstrap.servers=kafka:9094
group.id=mm-consumer-group
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
看看下面的producer.properties
代碼:
bootstrap.servers=10.1.1.10:9094
compression.type=none
從Docker堆棧運行命令:
kafka-mirror-maker.sh --consumer.config /opt/kafka/config/consumer.properties --num.streams 2 --producer.config /opt/kafka/config/producer.properties --whitelist=".*"
如果我在主機上運行相同的命令,則該命令有效。
更多日志輸出:
[2019-01-10 16:14:33,470] ERROR [mirrormaker-thread-0] Mirror maker thread failure due to (kafka.tools.MirrorMaker$MirrorMakerThread)
java.lang.IllegalStateException: No entry found for connection 2147482646
at org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:330)
at org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:134)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:885)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:276)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.tryConnect(ConsumerNetworkClient.java:548)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:655)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:635)
at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:575)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:389)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:231)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:316)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1214)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1179)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
at kafka.tools.MirrorMaker$ConsumerWrapper.receive(MirrorMaker.scala:481)
at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:370)
[2019-01-10 16:14:33,487] ERROR [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)
經過研究,該解決方案非常簡單,我必須在extra_hosts
docker-compose.yml
文件中添加帶有extra_hosts
的主機映射。
version: '3.6'
x-proxy: &proxy
http_proxy: ${http_proxy}
https_proxy: ${https_proxy}
no_proxy: ${no_proxy}
services:
mirror_maker:
image: wurstmeister/kafka:2.12-2.1.0
configs:
- source: mm-consumer-config
target: /opt/kafka/config/consumer.properties
- source: mm-producer-config
target: /opt/kafka/config/producer.properties
environment:
<<: *proxy
command: kafka-mirror-maker.sh --consumer.config /opt/kafka/config/consumer.properties --num.streams 2 --producer.config /opt/kafka/config/producer.properties --whitelist=.*
networks:
- workshop
extra_hosts:
- outside:10.1.1.10
- manager:10.1.1.11
restart: always
depends_on:
- zookeeper
- kafka
deploy:
# replicas: 2
placement:
constraints:
- node.role == manager
configs:
mm-consumer-config:
file: ./services/mirror_maker/config/consumer.properties
mm-producer-config:
file: ./services/mirror_maker/config/producer.properties
networks:
workshop:
name: workshop
external: true
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.