自动将在Kafka中创建的主题下沉到Elasticsearch

Question

I have topics being created in kafka (test1, test2, test3) and I want to sink them to elastic at creation time. I tried topics.regex but it only creates indices for topics already existing. How can I sink a new topic into an index when it gets created dynamically?

Here is the connector config that I am using for kafka-sink:

{
    "name": "elastic-sink-test-regex",
    "config": {
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "tasks.max": "1",
        "topics.regex": "test[0-9]+",
        "type.name": "kafka-connect",
        "connection.url": "http://192.168.0.188:9200",
        "key.ignore": "true",
        "schema.ignore": "true",
        "schema.enable": "false",
        "batch.size": "100",
        "flush.timeout.ms": "100000",
        "max.buffered.records": "10000",
        "max.retries": "10",
        "retry.backoff.ms": "1000",
        "max.in.flight.requests": "3",
        "is.timebased.indexed": "False",
        "time.index": "at"
    }
}

Answer 1

A sink connector won't read new topics till this connector is restarted (or a scheduled rebalance occurred). You can run a Kafka Stream that reads messages from new topics and put them into a result-like topic. A Sink Connector reads from the result-like topic.

To save a "message - topic" matching you can use Kafka Record Headers.

Make sure it meets your requirements!