如何在 KStream (Kafka Streams) 中加入列表的每个元素

Question

For example I have a KStream of shopping carts and each shopping cart has a List of product id's. Additionally there is a KStream with products. How do I join them together?

public class ShoppingCart {
    List<ProductKey> productKeys;
}

public class Product {
    ProductKey key;
    String name;
}

public class ProductKey {
    String id;
}

KStream<String, ShoppingCart> shoppingCartKStream;
KStream<ProductKey, Product> productKStream;

My desired result would look like this

KStream<String, ShoppingCartWithProducts> joinedStream;
public class ShoppingCartWithProducts {
    List<Product> products;
}

Is there an easy way to archive this?

EDIT: I know there is a way but I find it too complicated. To put it simply:

1. I need to flatMap the shoppingCart-KStream

2. Then I can join the result with the product-KStream

3. Group and aggregate the intermediate result

4. And finally join back with shoppingCart-KStream

    KStream<String, ProductKey> productKeyStream = shoppingCartKStream.flatMap((key, shoppingCart) -> shoppingCart.productKeys.stream().map(productKey -> KeyValue.pair(key, productKey)).collect(Collectors.toList()) ); KTable<String, Product> productStreamWithShoppingCartKey = productKeyStream.toTable().join( productKStream.toTable(), productKey -> productKey, (productKey, product) -> product ); KTable<String, ArrayList<Product>> productListStream = productStreamWithShoppingCartKey.groupBy(KeyValue::pair).aggregate( (Initializer<ArrayList<Product>>) ArrayList::new, (key, value, aggregate) -> addProductToList(aggregate, value), (key, value, aggregate) -> removeProductFromList(aggregate, value) ); KStream<String, ShoppingCartWithProducts> shoppingCartWithProductsKStream = shoppingCartKStream.join( productListStream, (shoppingCart, productList) -> new ShoppingCartWithProducts(productList) );

And of course it's very simplified, I also need to handle tombstone and so on.

Answer 1

After you define you StreamsBuilder that is the entry point to the Streams DSL.

StreamsBuilder builder = new StreamsBuilder();

you can do a join of a 5 minutes window using JoinWindows.of(Duration.ofMinutes(5)) . You have to have the keys of both stream using the same type otherwise kafka-stream cannot compare keys of different types. It is like a database join. So, I am using String for ShoppingCart and Product . Then the .join(... operator matches events of the same key and you can build your new event ShoppingCartWithProducts .

KStream<String, ShoppingCart> shoppingCartKStream = ...;
KStream<String, Product> productKStream = ...;

shoppingCartKStream.join(productKStream,
    (shop, prod) -> {
        log.info("ShoppingCart: {} with Product: {}", shop, prod);

        ShoppingCartWithProducts shoppingCartWithProducts = new ShoppingCartWithProducts();
        shoppingCartWithProducts.setShoppingCart(shop);
        shoppingCartWithProducts.setProduct(prod);
        return shoppingCartWithProducts;
    },
    JoinWindows.of(Duration.ofMinutes(5)),
    StreamJoined.with(Serdes.String(),
        new JsonSerde<>(ShoppingCart.class),
        new JsonSerde<>(Product.class)))
   .foreach((k, v) -> log.info("ShoppingCartWithProducts ID: {}, value: {}", k, v));

you can find more detailed information here .