[英]How to join multiple Kafka topics?
So I have...所以我有...
So a single logical business request can have multiple logs, commands and events associated to it by a uuid which the microservices pass to each other.因此,单个逻辑业务请求可以通过微服务相互传递的 uuid 关联多个日志、命令和事件。
So what are some of the technologies/patterns that can be used to read the 3 topics and join them all together as a single json document and then dump them to lets say Elasticsearch?那么,有哪些技术/模式可用于阅读这 3 个主题并将它们作为一个 json 文档连接在一起,然后将它们转储到 Elasticsearch?
Streaming?流媒体?
You can use Kafka Streams, or KSQL, to achieve this.您可以使用 Kafka Streams 或 KSQL 来实现这一点。 Which one depends on your preference/experience with Java, and also the specifics of the joins you want to do.哪一个取决于您对 Java 的偏好/经验,以及您想要执行的连接的具体细节。
KSQL is the SQL streaming engine for Apache Kafka, and with SQL alone you can declare stream processing applications against Kafka topics. KSQL是 Apache Kafka 的 SQL 流引擎,仅使用 SQL 就可以针对 Kafka 主题声明流处理应用程序。 You can filter, enrich, and aggregate topics.您可以过滤、丰富和聚合主题。 Currently only stream-table joins are supported.目前仅支持流表连接。 You can see an example in this article here您可以在此处查看本文中的示例
The Kafka Streams API is part of Apache Kafka, and a Java library that you can use to do stream processing of data in Apache Kafka. Kafka Streams API是 Apache Kafka 的一部分,是一个 Java 库,可用于在 Apache Kafka 中对数据进行流处理。 It is actually what KSQL is built on, and supports greater flexibility of processing, including stream-stream joins .它实际上是 KSQL 的基础,支持更大的处理灵活性,包括流-流连接。
You can use KSQL to join the streams.您可以使用 KSQL 加入流。
Benefits of using KSQL.使用 KSQL 的好处。
Drawback.退税。
Please have a look.请看一看。
https://github.com/confluentinc/ksql https://github.com/confluentinc/ksql
Same as question Is it possible to use multiple left join in Confluent KSQL query?同问题Is it possible to use multiple left join in Confluent KSQL query? tried to join stream with more than 1 tables, if not then whats the solution? 试图加入超过 1 个表的流,如果没有,那么解决方案是什么? And seems like you can not have multiple join keys within same query.并且似乎您不能在同一查询中拥有多个连接键。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.