[英]Kafka Streams not reading input topic
I have created sample Kafka Streams application from the tutorial:我从教程中创建了示例 Kafka Streams 应用程序:
public static void main(String[] args) throws Exception {
Logger log = Logger.getLogger("Name");
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-wordprint");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.99.100:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
final KStreamBuilder builder = new KStreamBuilder();
builder.stream("onecon_postgres").print();
final KafkaStreams streams = new KafkaStreams(builder, props);
final CountDownLatch latch = new CountDownLatch(1);
// attach shutdown handler to catch control-c
Runtime.getRuntime().addShutdownHook(new Thread("streams-shutdown-hook") {
@Override
public void run() {
streams.close();
latch.countDown();
}
});
try {
streams.start();
log.info("After Start");
latch.await();
} catch (Throwable e) {
System.exit(1);
}
System.exit(0);
}
Unfortunately this application does not read input stream.不幸的是,此应用程序不读取输入流。 I have a JDBC source connector from PostgreSQL and it's working fine streaming data from one database (I can see on Kafka Connect UI data within this topic).我有一个来自 PostgreSQL 的 JDBC 源连接器,它可以很好地处理来自一个数据库的流数据(我可以在本主题中的 Kafka Connect UI 数据上看到)。
The problem I have is even though I have changed IP in BOOTSTRAP_SERVERS_CONFIG in Properties IP is localhost I don't know why.我遇到的问题是,即使我在 Properties IP 中的BOOTSTRAP_SERVERS_CONFIG中更改了 IP 是localhost我不知道为什么。
[main] INFO org.apache.kafka.streams.StreamsConfig - StreamsConfig values:
application.id = streams-linesplit
application.server =
**bootstrap.servers = [localhost:9092]**
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 10485760
client.id =
commit.interval.ms = 30000
connections.max.idle.ms = 540000
default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
default.timestamp.extractor = class org.apache.kafka.streams.processor.FailOnInvalidTimestamp
default.value.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
key.serde = null
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
num.standby.replicas = 0
num.stream.threads = 1
partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 40000
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 600000
state.dir = /tmp/kafka-streams
timestamp.extractor = null
value.serde = null
windowstore.changelog.additional.retention.ms = 86400000
zookeeper.connect =
To overcome this I have used netsh to forward traffic but I cannot see this application to consume my stream.为了克服这个问题,我使用了 netsh 来转发流量,但我看不到这个应用程序来使用我的流。
netsh interface portproxy add v4tov4 listenport=9092 listenaddress=127.0.0.1 connectport=9092 connectaddress=192.168.99.100
Unfortunately this application does not read input stream.不幸的是,此应用程序不读取输入流。
You seem to have a networking problem between your Kafka Streams application and your Kafka broker.您的 Kafka Streams 应用程序和 Kafka 代理之间似乎存在网络问题。 It is rather unlikely that "Kafka Streams does not work". “Kafka Streams 不起作用”的可能性很小。
Also, it's hard to help you without you providing more information:此外,如果您不提供更多信息,很难为您提供帮助:
192.168.99.100:9092
?是192.168.99.100:9092
吗?ERROR
or WARN
log messages?您是否看到ERROR
或WARN
日志消息?The problem I have is even though I have changed IP in BOOTSTRAP_SERVERS_CONFIG in Properties IP is localhost I don't know why.我遇到的问题是,即使我在 Properties IP 中的 BOOTSTRAP_SERVERS_CONFIG 中更改了 IP 是 localhost 我不知道为什么。
I don't understand -- why do you think changing the BOOTSTRAP_SERVERS_CONFIG
to localhost:9092
will fix your original problem?我不明白——为什么你认为将BOOTSTRAP_SERVERS_CONFIG
更改为localhost:9092
会解决你原来的问题? I understood that the Kafka broker actually listens on 192.168.99.100:9092
?我知道 Kafka 经纪人实际上在192.168.99.100:9092
侦听?
To overcome this I have used netsh to forward traffic but I cannot see this application to consume my stream.为了克服这个问题,我使用了 netsh 来转发流量,但我看不到这个应用程序来使用我的流。
The port forwarding will most probably not help.端口转发很可能无济于事。 Without updating the configuration of your Kafka broker, the broker will by default only communicate on its "real" IP + port.在不更新 Kafka 代理的配置的情况下,代理默认只会在其“真实”IP + 端口上进行通信。 Slightly simplified: the broker, configured to listen on 192.168.99.100:9092
, will not respond to localhost:9092
request that your Kafka Streams application sends, even though you are doing port forwarding from localhost:9092 -> 192.168.99.100:9092
on the machine that runs your Kafka Streams application.稍微简化:配置为侦听192.168.99.100:9092
的代理不会响应您的 Kafka Streams 应用程序发送的localhost:9092
请求,即使您正在从localhost:9092 -> 192.168.99.100:9092
进行端口转发运行 Kafka Streams 应用程序的机器。
Hope this helps a bit!希望这个对你有帮助!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.