如何从外部访问连接到路由器的 Kafka Multicluster？

Question

我的母语不是英语。

所以我很抱歉我的英语水平不好。

我必须构建一个 Kafka 多集群。

所以我用三台计算机构建了一个Kafka Multicluster，就像这样

我在路由器上设置了端口转发，因为路由器连接了三台计算机。

220.***.***.7:9091 -> 192.168.0.201:9092 (cluster 1 kafka port)
220.***.***.7:9092 -> 192.168.0.202:9092 (cluster 2 kafka port)
220.***.***.7:9093 -> 192.168.0.203:9092 (cluster 3 kafka port)

220.***.***.7:2181 -> 192.168.0.201:2181 (cluster 1 zoo-keeper port)
220.***.***.7:2182 -> 192.168.0.202:2181 (cluster 2 zoo-keeper port)
220.***.***.7:2183 -> 192.168.0.203:2181 (cluster 3 zoo-keeper port)

路由器内部的计算机通常可以获取值，但路由器外部的计算机甚至无法访问 kafka。

这是我的 kafka 和 zookeeper 设置。

...
############################# cluster 1's servers.properties #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.0.201:9092
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181

############################# Socket Server Settings #############################
...

...
#cluster 1's /etc/hosts
192.168.0.201 kafka01
192.168.0.202 kafka02
192.168.0.203 kafka03
...

...
#cluster 1's zookeeper.properties
initLimit=5000
syncLimit=2000

server.1=0.0.0.0:2888:3888
server.2=kafka02:2888:3888
server.3=kafka03:2888:3888
...

...
############################# cluster 2's servers.properties #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.0.202:9092
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181

############################# Socket Server Settings #############################
...

...
#cluster 2's /etc/hosts
192.168.0.201 kafka01
192.168.0.202 kafka02
192.168.0.203 kafka03
...

...
#cluster 2's zookeeper.properties
initLimit=5000
syncLimit=2000

server.1=kafka01:2888:3888
server.2=0.0.0.0:2888:3888
server.3=kafka03:2888:3888
...

...
############################# cluster 3's servers.properties #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=3
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://192.168.0.203:9092
zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181

############################# Socket Server Settings #############################
...

...
#cluster 3's /etc/hosts
192.168.0.201 kafka01
192.168.0.202 kafka02
192.168.0.203 kafka03
...

...
#cluster 3's zookeeper.properties
initLimit=5000
syncLimit=2000

server.1=kafka01:2888:3888
server.2=kafka02:2888:3888
server.3=0.0.0.0:2888:3888
...

当我在路由器 A 中运行下面的 Python 代码时，我正常收到了测试主题的值。

from kafka import KafkaConsumer
bootstrap_servers = ["220.***.***.7:9091","220.***.***.7:9092","220.***.***.7:9093"]
consumer = KafkaConsumer("test",bootstrap_servers=bootstrap_servers,auto_offset_reset = "earliest",group_id="123")

for i in consumer:
    print(i)

但是，如果我在路由器之外执行相同的代码，我会收到“nobrokersavailable”错误。

所以我用下面的代码做了调试。

from kafka import KafkaConsumer
import logging #ADDED CODE
logging.basicConfig(level=logging.DEBUG) #ADDED CODE

bootstrap_servers = ["220.***.***.7:9091","220.***.***.7:9092","220.***.***.7:9093"]
consumer = KafkaConsumer("test",bootstrap_servers=bootstrap_servers,auto_offset_reset = "earliest",group_id="12356") #I changed group_id

for i in consumer:
    print(i)

然后，它在下面的调试代码中显示。

DEBUG:kafka.metrics.metrics:Added sensor with name connections-closed
DEBUG:kafka.metrics.metrics:Added sensor with name connections-created
DEBUG:kafka.metrics.metrics:Added sensor with name select-time
DEBUG:kafka.metrics.metrics:Added sensor with name io-time
DEBUG:kafka.client:Initiating connection to node bootstrap-0 at 220.***.***.7:9091
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent-received
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name request-latency
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-0.latency
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <disconnected> [IPv4 None]>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <disconnected> [IPv4 ('220.***.***.7', 9091)]>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connecting> [IPv4 ('220.***.***.7', 9091)]>: connecting to 220.***.***.7:9091 [('220.***.***.7', 9091) IPv4]
INFO:kafka.conn:Probing node bootstrap-0 broker version
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connecting> [IPv4 ('220.***.***.7', 9091)]>: established TCP connection
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connecting> [IPv4 ('220.***.***.7', 9091)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-0 connected
DEBUG:kafka.protocol.parser:Sending request ApiVersionRequest_v0()
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Request 1: ApiVersionRequest_v0()
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v0(topics=[])
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Request 2: MetadataRequest_v0(topics=[])
DEBUG:kafka.protocol.parser:Received correlation id: 1
DEBUG:kafka.protocol.parser:Processing response ApiVersionResponse_v0
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Response 1 (100.98743438720703 ms): ApiVersionResponse_v0(error_code=0, api_versions=[(api_key=0, min_version=0, max_version=8), (api_key=1, min_version=0, max_version=12), (api_key=2, min_version=0, max_version=5), (api_key=3, min_version=0, max_version=9), (api_key=4, min_version=0, max_version=4), (api_key=5, min_version=0, max_version=3), (api_key=6, min_version=0, max_version=6), (api_key=7, min_version=0, max_version=3), (api_key=8, min_version=0, max_version=8), (api_key=9, min_version=0, max_version=7), (api_key=10, min_version=0, max_version=3), (api_key=11, min_version=0, max_version=7), (api_key=12, min_version=0, max_version=4), (api_key=13, min_version=0, max_version=4), (api_key=14, min_version=0, max_version=5), (api_key=15, min_version=0, max_version=5), (api_key=16, min_version=0, max_version=4), (api_key=17, min_version=0, max_version=1), (api_key=18, min_version=0, max_version=3), (api_key=19, min_version=0, max_version=6), (api_key=20, min_version=0, max_version=5), (api_key=21, min_version=0, max_version=2), (api_key=22, min_version=0, max_version=4), (api_key=23, min_version=0, max_version=3), (api_key=24, min_version=0, max_version=2), (api_key=25, min_version=0, max_version=2), (api_key=26, min_version=0, max_version=2), (api_key=27, min_version=0, max_version=0), (api_key=28, min_version=0, max_version=3), (api_key=29, min_version=0, max_version=2), (api_key=30, min_version=0, max_version=2), (api_key=31, min_version=0, max_version=2), (api_key=32, min_version=0, max_version=3), (api_key=33, min_version=0, max_version=1), (api_key=34, min_version=0, max_version=1), (api_key=35, min_version=0, max_version=2), (api_key=36, min_version=0, max_version=2), (api_key=37, min_version=0, max_version=3), (api_key=38, min_version=0, max_version=2), (api_key=39, min_version=0, max_version=2), (api_key=40, min_version=0, max_version=2), (api_key=41, min_version=0, max_version=2), (api_key=42, min_version=0, max_version=2), (api_key=43, min_version=0, max_version=2), (api_key=44, min_version=0, max_version=1), (api_key=45, min_version=0, max_version=0), (api_key=46, min_version=0, max_version=0), (api_key=47, min_version=0, max_version=0), (api_key=48, min_version=0, max_version=0), (api_key=49, min_version=0, max_version=0), (api_key=50, min_version=0, max_version=0), (api_key=51, min_version=0, max_version=0), (api_key=56, min_version=0, max_version=0), (api_key=57, min_version=0, max_version=0)])
DEBUG:kafka.protocol.parser:Received correlation id: 2
DEBUG:kafka.protocol.parser:Processing response MetadataResponse_v0
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Response 2 (3.607034683227539 ms): MetadataResponse_v0(brokers=[(node_id=2, host='192.168.0.202', port=9092), (node_id=3, host='192.168.0.203', port=9092), (node_id=1, host='192.168.0.201', port=9092)], topics=[(error_code=0, topic='hanasd', partitions=[(error_code=0, partition=0, leader=3, replicas=[3, 2], isr=[2, 3])]), (error_code=0, topic='test', partitions=[(error_code=0, partition=0, leader=2, replicas=[1, 2, 3], isr=[2, 1, 3]), (error_code=0, partition=2, leader=3, replicas=[3, 1, 2], isr=[2, 1, 3]), (error_code=0, partition=1, leader=2, replicas=[2, 3, 1], isr=[2, 1, 3])]), (error_code=0, topic='__consumer_offsets', partitions=[(error_code=0, partition=0, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=10, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=20, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=40, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=30, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=9, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=11, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=31, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=39, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=13, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=18, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=22, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=32, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=8, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=43, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=29, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=34, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=1, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=6, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=41, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=27, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=48, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=5, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=15, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=35, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=25, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=46, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=26, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=36, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=44, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=4, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=16, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=17, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=3, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=45, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=37, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=24, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=38, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=33, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=23, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=28, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=2, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=12, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=19, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=14, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=47, leader=2, replicas=[2], isr=[2]), (error_code=0, partition=49, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=42, leader=3, replicas=[3], isr=[3]), (error_code=0, partition=7, leader=1, replicas=[1], isr=[1]), (error_code=0, partition=21, leader=3, replicas=[3], isr=[3])])])
INFO:kafka.conn:Broker version identified as 2.5.0
INFO:kafka.conn:Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
DEBUG:kafka.metrics.metrics:Added sensor with name bytes-fetched
DEBUG:kafka.metrics.metrics:Added sensor with name records-fetched
DEBUG:kafka.metrics.metrics:Added sensor with name fetch-latency
DEBUG:kafka.metrics.metrics:Added sensor with name records-lag
DEBUG:kafka.metrics.metrics:Added sensor with name fetch-throttle-time
DEBUG:kafka.metrics.metrics:Added sensor with name heartbeat-latency
DEBUG:kafka.metrics.metrics:Added sensor with name join-latency
DEBUG:kafka.metrics.metrics:Added sensor with name sync-latency
DEBUG:kafka.metrics.metrics:Added sensor with name commit-latency
INFO:kafka.consumer.subscription_state:Updating subscribed topics to: ('test',)
DEBUG:kafka.coordinator:Sending group coordinator request for group 123456 to broker bootstrap-0
DEBUG:kafka.protocol.parser:Sending request GroupCoordinatorRequest_v0(consumer_group='123456')
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Request 3: GroupCoordinatorRequest_v0(consumer_group='123456')
DEBUG:kafka.client:Initializing connection to node bootstrap-2 for metadata request
DEBUG:kafka.client:Initiating connection to node bootstrap-2 at 220.***.***.7:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-2.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-2.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-bootstrap-2.latency
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <disconnected> [IPv4 None]>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <disconnected> [IPv4 ('220.***.***.7', 9092)]>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <connecting> [IPv4 ('220.***.***.7', 9092)]>: connecting to 220.***.***.7:9092 [('220.***.***.7', 9092) IPv4]
DEBUG:kafka.protocol.parser:Received correlation id: 3
DEBUG:kafka.protocol.parser:Processing response GroupCoordinatorResponse_v0
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=220.***.***.7:9091 <connected> [IPv4 ('220.***.***.7', 9091)]> Response 3 (1.6961097717285156 ms): GroupCoordinatorResponse_v0(error_code=0, coordinator_id=3, host='192.168.0.203', port=9092)
DEBUG:kafka.coordinator:Received group coordinator response GroupCoordinatorResponse_v0(error_code=0, coordinator_id=3, host='192.168.0.203', port=9092)
DEBUG:kafka.cluster:Updating coordinator for 123456: GroupCoordinatorResponse_v0(error_code=0, coordinator_id=3, host='192.168.0.203', port=9092)
INFO:kafka.cluster:Group coordinator for 123456 is BrokerMetadata(nodeId='coordinator-3', host='192.168.0.203', port=9092, rack=None)
INFO:kafka.coordinator:Discovered coordinator coordinator-3 for group 123456
INFO:kafka.coordinator:Starting new heartbeat thread
DEBUG:kafka.coordinator:Heartbeat thread started
INFO:kafka.coordinator.consumer:Revoking previously assigned partitions set() for group 123456
DEBUG:kafka.client:Initiating connection to node coordinator-3 at 192.168.0.203:9092
DEBUG:kafka.metrics.metrics:Added sensor with name node-coordinator-3.bytes-sent
DEBUG:kafka.metrics.metrics:Added sensor with name node-coordinator-3.bytes-received
DEBUG:kafka.metrics.metrics:Added sensor with name node-coordinator-3.latency
DEBUG:kafka.conn:<BrokerConnection node_id=coordinator-3 host=192.168.0.203:9092 <disconnected> [IPv4 None]>: creating new socket
DEBUG:kafka.conn:<BrokerConnection node_id=coordinator-3 host=192.168.0.203:9092 <disconnected> [IPv4 ('192.168.0.203', 9092)]>: setting socket option (6, 1, 1)
INFO:kafka.conn:<BrokerConnection node_id=coordinator-3 host=192.168.0.203:9092 <connecting> [IPv4 ('192.168.0.203', 9092)]>: connecting to 192.168.0.203:9092 [('192.168.0.203', 9092) IPv4]
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <connecting> [IPv4 ('220.***.***.7', 9092)]>: established TCP connection
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <connecting> [IPv4 ('220.***.***.7', 9092)]>: Connection complete.
DEBUG:kafka.client:Node bootstrap-2 connected
DEBUG:kafka.client:Sending metadata request MetadataRequest_v1(topics=['test']) to node bootstrap-2
DEBUG:kafka.protocol.parser:Sending request MetadataRequest_v1(topics=['test'])
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <connected> [IPv4 ('220.***.***.7', 9092)]> Request 1: MetadataRequest_v1(topics=['test'])
DEBUG:kafka.protocol.parser:Received correlation id: 1
DEBUG:kafka.protocol.parser:Processing response MetadataResponse_v1
DEBUG:kafka.conn:<BrokerConnection node_id=bootstrap-2 host=220.***.***.7:9092 <connected> [IPv4 ('220.***.***.7', 9092)]> Response 1 (100.69775581359863 ms): MetadataResponse_v1(brokers=[(node_id=2, host='192.168.0.202', port=9092, rack=None), (node_id=3, host='192.168.0.203', port=9092, rack=None), (node_id=1, host='192.168.0.201', port=9092, rack=None)], controller_id=1, topics=[(error_code=0, topic='test', is_internal=False, partitions=[(error_code=0, partition=0, leader=2, replicas=[1, 2, 3], isr=[2, 1, 3]), (error_code=0, partition=2, leader=3, replicas=[3, 1, 2], isr=[2, 1, 3]), (error_code=0, partition=1, leader=2, replicas=[2, 3, 1], isr=[2, 1, 3])])])
DEBUG:kafka.cluster:Updated cluster metadata to ClusterMetadata(brokers: 3, topics: 1, groups: 1)

那么，如何从外部访问连接到路由器的 Kafka Multicluster？

Answer 1

注意在 output 你得到MetadataResponse_v0(brokers=[(node_id=2, host='192.168.0.202', port=9092 ，所以端口转发工作，但它连接到唯一的广告地址

除了内部 LAN 地址之外，您还需要宣传您的 220.xx7 地址，但是，您不应该将纯文本 Kafka 暴露给外部互联网

此外，您不需要使用 hosts 文件，因为您的路由器可能具有功能正常的 DNS 服务器，并且您不必转发 Zookeeper