从Kafka流到Elasticsearch时的主题映射

Question

When I transfer or stream two and three tables then I can easily map in Elasticsearch but can I map automatically map topics to index

I have streamed data from PostgreSQL to ES by mapping manually topic.index.map=topic1:index1,topic2:index2 , etc.

Can I map automatically whatever topics send by producer then consumer consume in ES connector automatically?

Answer 1

By default, the topics map directly to an index of the same name.

If you want "better" control, you can use RegexRouter in a transforms property

To quote the docs

topic.index.map

This option is now deprecated. A future version may remove it completely. Please use single message transforms, such as RegexRouter , to map topic names to index names

If you cannot capture a single regex for each topic in the connector, then run more connectors with a different pattern