[英]How to join two Kafka streams and produce the result in a topic with Avro values
I've got two Kafka Streams with keys in String
and values in Avro
format which I have created using KSQL.我有两个 Kafka Streams,键是String
,值是Avro
格式,这是我使用 KSQL 创建的。
Here's the first one:这是第一个:
DESCRIBE EXTENDED STREAM_1;
Type : STREAM
Key field : IDUSER
Timestamp field : Not set - using <ROWTIME>
Key format : STRING
Value format : AVRO
Kafka output topic : STREAM_1 (partitions: 4, replication: 1)
Field | Type
--------------------------------------------------------
ROWTIME | BIGINT (system)
ROWKEY | VARCHAR(STRING) (system)
FIRSTNAME | VARCHAR(STRING)
LASTNAME | VARCHAR(STRING)
IDUSER | VARCHAR(STRING)
and the second one:第二个:
DESCRIBE EXTENDED STREAM_2;
Type : STREAM
Key field : IDUSER
Timestamp field : Not set - using <ROWTIME>
Key format : STRING
Value format : AVRO
Kafka output topic : STREAM_2 (partitions: 4, replication: 1)
Field | Type
--------------------------------------------------------
ROWTIME | BIGINT (system)
ROWKEY | VARCHAR(STRING) (system)
USERNAME | VARCHAR(STRING)
IDUSER | VARCHAR(STRING)
DEVICE | VARCHAR(STRING)
The desired output should include IDUSER
, LASTNAME
, DEVICE
and USERNAME
.所需的输出应包括IDUSER
、 LASTNAME
、 DEVICE
和USERNAME
。
I want to left join
these streams (on IDUSER
) using Streams API and write the output into a kafka topic.我想使用 Streams API left join
这些流(在IDUSER
)并将输出写入 kafka 主题。
To do so, I've tried the following:为此,我尝试了以下方法:
public static void main(String[] args) {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "kafka-strteams");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "localhost:2181");
streamsConfiguration.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081");
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, GenericAvroSerde.class);
streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
final Serde<String> stringSerde = Serdes.String();
final Serde<GenericRecord> genericAvroSerde = new GenericAvroSerde();
boolean isKeySerde = false;
genericAvroSerde.configure(Collections.singletonMap(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081"), isKeySerde);
KStreamBuilder builder = new KStreamBuilder();
KStream<String, GenericRecord> left = builder.stream("STREAM_1");
KStream<String, GenericRecord> right = builder.stram("STREAM_2");
// Java 8+ example, using lambda expressions
KStream<String, GenericRecord> joined = left.leftJoin(right,
(leftValue, rightValue) -> "left=" + leftValue + ", right=" + rightValue, /* ValueJoiner */
JoinWindows.of(TimeUnit.MINUTES.toMillis(5)),
Joined.with(
stringSerde, /* key */
genericAvroSerde, /* left value */
genericAvroSerde) /* right value */
);
joined.to(stringSerde, genericAvroSerde, "streams-output-testing");
KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.cleanUp();
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
However,然而,
KStream<String, GenericRecord> joined = ...
throws an error on my IDE:在我的 IDE 上引发错误:
incompatible types: inference variable VR has incompatible bounds
When I try to use a String
Serde
for both keys and values, it works but the data is not that readable from kafka-console-consumer
.当我尝试对键和值使用String
Serde
,它可以工作,但数据不是从kafka-console-consumer
读取的。 What I want to do is to produce the data in AVRO format in order to be able to read them off using kafka-avro-console-consumer
.我想要做的是以 AVRO 格式生成数据,以便能够使用kafka-avro-console-consumer
读取它们。
My first guess is that you are returning a String
from the join operation, whereas your code expects a GenericRecord
as the result:我的第一个猜测是您从连接操作返回一个String
,而您的代码期望一个GenericRecord
作为结果:
KStream<String, GenericRecord> joined = left.leftJoin(right,
(leftValue, rightValue) -> "left=" + leftValue + ", right=" + rightValue, ...)
Note how joined
has type KStream<String, GenericRecord>
, ie the value has type GenericRecord
, but the join output is computed via "left=" + leftValue + ", right=" + rightValue
, which has type String
.注意如何joined
具有式KStream<String, GenericRecord>
即,值具有类型GenericRecord
,但输出经由计算的加入"left=" + leftValue + ", right=" + rightValue
,其具有类型String
。
Instead of converting value into string you can directly return value.您可以直接返回值,而不是将值转换为字符串。 For example :例如 :
KStream joined = left.leftJoin(right,
(leftValue, rightValue) -> { return rightValue});
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.