如何根据来自另一个主题的响应读取来自 Kafka 主题的消息

Question

I have 2 topics in Kafka: MetaData and MasterData. And I am reading data in real time using Kafka Listners. There are 2 scenarios:

1. I have to read MetaData topic first and then MasterData topic.
2. There could also be a possibility that no new msg comes in MetaData topic, but a message is inserted in MasterData topic. In such as case consumption from MasterData topic should go ahead.

Ant suggestions how to achieve this??

Answer 1

After the clarification in the comments this is, what I understand, what you want to achieve:

You have a database like system, and the meta data from the kafka topic is used to adjust the table structure. Then the actual data comes into the master kafka topic which you then want to insert into the table. In other words: You want to define the structure of your data using kafka.

I am not sure, if that's what I would call a regular use case for Kafka, as it is meant to be an interface to exchange events, aka data. What you try to do, is to use the system as a means of defining data structure. But that's only my opinion.

As said earlier in the comments, there is practically no way to say that there won't be a message in one topic. You can only say: there is no message "yet". What you "could" probably do is: When a message in the master topic arrives, you first check the meta topic, if there is a message, too. If there is, you apply the changes in data structure, then import the message fron the master topic.

Another option would be to use Kafka Streams instead of the raw consumers to bring together the two topics, with a join for example.

Anyway to define the data structure, one would normally use something like Avro, which gives you schema evolution and more. Then write your application to be aware of schema changes an apply them to your database tables accordingly.

UPDATE: How to solve using Kafka Streams

As mentioned above, the use of Kafka Streams would be a possible solution to your case. Let me explain, what i mean in some pseudo kafka streams code:

import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.GlobalKTable;

import java.nio.charset.StandardCharsets;

/**
 * Just an incomplete Kafka Streams Code Demo to show how the problem could
 * be solved with this framework instead of using the consumers directly.
 * Kafka Streams Boilerplate code not included.
 */
public class Example {
    private static final String META_TOPIC = "meta";
    private static final String MASTER_TOPIC = "master";

    // You have to make sure, the meta data is stored under this
    // specific key in the meta topic.
    private static final byte[] META_KEY = MetaEvent.class.getName().getBytes(StandardCharsets.UTF_8);

    public static void main(String[] args) {
        new Example().createTopology();
    }

    public void createTopology() {

        final StreamsBuilder builder = new StreamsBuilder();

        final GlobalKTable<byte[], MetaEvent> metaTable = builder.globalTable(META_TOPIC);

        builder.<byte[], MasterEvent>stream(MASTER_TOPIC)
                .leftJoin(
                        metaTable,
                        (k, v) -> META_KEY,
                        MasterWithMeta::new)
                .foreach(this::handleEvent);
    }

    private void handleEvent(byte[] key, MasterWithMeta masterWithMeta) {
        // 1) check if meta has changed, if so, apply changes to database
        // 2) import master data to database
    }
}

class MasterWithMeta {
    private final MasterEvent master;
    private final MetaEvent meta;

    public MasterEvent getMaster() {
        return master;
    }

    public MetaEvent getMeta() {
        return meta;
    }

    public MasterWithMeta(MasterEvent master, MetaEvent meta) {
        this.master = master;
        this.meta = meta;
    }

    public static MasterWithMeta create(MasterEvent master, MetaEvent meta) {
        return new MasterWithMeta(master, meta);
    }
}

class MetaEvent {
    // ...
}

class MasterEvent {
    // ...
}