Kafka stream 消費者 JSON object：如何 Z1D78DC8ED51214E518B5114AE2444

Question

我是 Kafka/Kafka Stream 的新手。 我正在使用最新的 Kafka/kafka-stream 和 kafka-client 和openjdk11 。 我的制片人正在生產 json 對象（其中鍵是名稱），看起來像

{"Name":"John", "amount":123, "time":2019-10-03T05:24:52" }

生產者代碼以便更好地理解：

public static ProducerRecord<String, String> newRandomTransaction(String name) {
    // creates an empty json {}
    ObjectNode transaction = JsonNodeFactory.instance.objectNode();

    Integer amount = ThreadLocalRandom.current().nextInt(0, 100);

    // Instant.now() is to get the current time
    Instant now = Instant.now();

    // we write the data to the json document
    transaction.put("name", name);
    transaction.put("amount", amount);
    transaction.put("time", now.toString());
    return new ProducerRecord<>("bank-transactions", name, transaction.toString());
}

現在我正在嘗試編寫我的應用程序來消耗交易並計算該人余額中的總金額。

（僅供參考：我正在使用舊代碼並試圖使其工作）。

使用GroupBYKey作為主題已經有正確的鍵。 然后匯總以計算我正在努力的總余額。

此時的應用程序（注釋掉的部分是我試圖使其在下一行中工作的舊代碼）：

public class BankBalanceExactlyOnceApp {
    private static ObjectMapper mapper = new ObjectMapper();

    public static void main(String[] args) {
        Properties config = new Properties();

        config.put(StreamsConfig.APPLICATION_ID_CONFIG, "bank-balance-application");
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
        config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        // we disable the cache to demonstrate all the "steps" involved in the transformation - not recommended in prod
        config.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, "0");

        // Exactly once processing!!
        config.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE);

        // json Serde
        final Serializer<JsonNode> jsonSerializer = new JsonSerializer();
        final Deserializer<JsonNode> jsonDeserializer = new JsonDeserializer();
        final Serde<JsonNode> jsonSerde = Serdes.serdeFrom(jsonSerializer, jsonDeserializer);


        StreamsBuilder builder = new StreamsBuilder();

        KStream<String, JsonNode> bankTransactions =
                builder.stream( "bank-transactions", Materialized.with(Serdes.String(), jsonSerde);


        // create the initial json object for balances
        ObjectNode initialBalance = JsonNodeFactory.instance.objectNode();
        initialBalance.put("count", 0);
        initialBalance.put("balance", 0);
        initialBalance.put("time", Instant.ofEpochMilli(0L).toString());

        /*KTable<String, JsonNode> bankBalance = bankTransactions
                .groupByKey(Serdes.String(), jsonSerde)
                .aggregate(
                        () -> initialBalance,
                        (key, transaction, balance) -> newBalance(transaction, balance),
                        jsonSerde,
                        "bank-balance-agg"
                );*/

        KTable<String, JsonNode> bankBalance = bankTransactions
                .groupByKey(Serialized.with(Serdes.String(), jsonSerde))
                .aggregate(
                        () -> initialBalance,
                        (key, transaction, balance) -> {
                            //String t = transaction.toString();
                            newBalance(transaction, balance);
                        },
                        Materialized.with(Serdes.String(), jsonSerde),
                        "bank-balance-agg"
                );

        bankBalance.toStream().to("bank-balance-exactly-once", Produced.with(Serdes.String(), jsonSerde));

        KafkaStreams streams = new KafkaStreams(builder.build(), config);
        streams.cleanUp();
        streams.start();

        // print the topology
        System.out.println(streams.toString());

        // shutdown hook to correctly close the streams application
        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }

    private static JsonNode newBalance(JsonNode transaction, JsonNode balance) {
        // create a new balance json object
        ObjectNode newBalance = JsonNodeFactory.instance.objectNode();
        newBalance.put("count", balance.get("count").asInt() + 1);
        newBalance.put("balance", balance.get("balance").asInt() + transaction.get("amount").asInt());

        Long balanceEpoch = Instant.parse(balance.get("time").asText()).toEpochMilli();
        Long transactionEpoch = Instant.parse(transaction.get("time").asText()).toEpochMilli();
        Instant newBalanceInstant = Instant.ofEpochMilli(Math.max(balanceEpoch, transactionEpoch));
        newBalance.put("time", newBalanceInstant.toString());
        return newBalance;
    }
}

問題是：當我嘗試在該行中調用newBalance(transaction, balance)時：

aggregate(
                    () -> initialBalance,
                    (key, transaction, balance) -> newBalance(transaction, balance),
                    jsonSerde,
                    "bank-balance-agg"
            )

並使用 msg 查看編譯器錯誤：

newBalance(JsonNode, JsonNode) can not be applied to (<lambda parameter>,<lambda parameter>)

我嘗試將其讀取為字符串，將參數類型從 JsonNode 更改為 Object。 但是，無法修復它。

我可以就如何修復它獲得任何建議嗎？

Answer 1

Kafka Streams 2.3 中的KGroupedStream沒有具有以下簽名的方法：

<VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
                             final Aggregator<? super K, ? super V, VR> aggregator,
                             final Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized,
                             String aggregateName);

有兩個重載方法aggregate ：

<VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
                             final Aggregator<? super K, ? super V, VR> aggregator);

<VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
                             final Aggregator<? super K, ? super V, VR> aggregator,
                             final Materialized<K, VR, KeyValueStore<Bytes, byte[]>> materialized);

您應該使用第二個，您的代碼應該類似於：

KTable<String, JsonNode> bankBalance = input
        .groupByKey(Grouped.with(Serdes.String(), jsonSerde))
        .aggregate(
                () -> initialBalance,
                (key, transaction, balance) -> newBalance(transaction, balance),
                Materialized.with(Serdes.String(), jsonSerde)
        );