通過 Azure 上的 Jumphost 連接到 Kafka 集群的問題

Question

我正在努力解決這個問題。 我正在嘗試通過 jumphost 連接連接到 kafka 集群：為此，我使用的是 az cli + ssh：

az ssh vm -n jumphost -g PROD-HUB -- -vvv -L 9093:kafka.dev:9093

我在某處讀到我應該 map 與/etc/hosts.conf中的本地主機相同的 kafka 主機名稱

127.0.0.1   kafka.dev

上面的連接完美無缺，我可以連接到轉發端口：

curl -v telnet://kafka.dev:9093
*   Trying 127.0.0.1:9093...
* TCP_NODELAY set
* Connected to kafka.dev (127.0.0.1) port 9093 (#0)

我在下面這樣配置了監聽器和adverted.listeners：

listeners=INTERNAL://kafka.dev:9093
advertised.listeners=INTERNAL://kafka.dev:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT

但是當我嘗試使用以下命令通過kafkacat進行調試時：

kafkacat -L -b kafka.dev:9093 -d broker

我收到錯誤：

%7|1660324800.333|BROKER|rdkafka#producer-1| [thrd:app]: kafka.dev:9093/bootstrap: Added new broker with NodeId -1
%7|1660324800.333|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: kafka.dev:9093/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1660324800.333|BRKMAIN|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Enter main broker thread
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Received CONNECT op
%7|1660324800.333|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.2.1 (0x10201ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL SASL_CYRUS HDRHISTOGRAM LZ4_EXT SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1660324800.333|STATE|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: broker in state TRY_CONNECT connecting
%7|1660324800.333|STATE|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Connecting to ipv4#127.0.0.1:9093 (plaintext) with socket 7
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Connected to ipv4#127.0.0.1:9093
%7|1660324800.333|CONNECTED|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Connected (#1)
%7|1660324800.333|FEATURE|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1660324800.333|STATE|rdkafka#producer-1| [thrd:kafka.dev:9093/bootstrap]: kafka.dev:9093/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324800.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324801.333|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
^C%7|1660324802.333|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
^C^C%7|1660324803.333|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1660324804.333|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1660324805.333|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1660324805.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1660324805.333|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
% ERROR: Failed to acquire metadata: Local: Broker transport failure

我很想在這個話題上得到一些幫助。 我已經為此苦苦掙扎了 2 天:)

Answer 1

對於顯示的配置，Kafka 需要在跳轉主機本身上運行。

這樣，您的/etc/hosts指向kafka.dev DNS 名稱的127.0.0.1 ，該名稱與advertised.listeners匹配。 當 Kafka 客戶端解析kafka.dev時，它將指向本地機器，而不是在 Azure 中運行的任何外部 Kafka 集群。

一般來說，很難通過代理/“跳轉主機”連接到 Kafka，因為客戶端必須連接到各個代理，並且必須可以通過配置的advertised.listeners設置由任何客戶端解析。 而且，不，應該避免編輯/etc/hosts ，因為 Kafka 不需要任何 DNS 名稱配置。

如果您想 SSH 代理 Kafka 連接，則需要所有代理來通告代理的 SSH 端口，該端口不需要匹配綁定listeners器配置中的任何端口，但是，如果您在advertised.listeners中設置了一個端口不在listeners中，則服務器將拒絕廣告端口上的任何連接，因為沒有為其打開的服務器端口（僅由listeners配置）。

總而言之，您可能更容易配置 Azure EventHubs，例如，然后將網絡策略設置為根本不需要跳轉框。