我可以让Kafka使用者/接收器连接以跳过主题中的特定分区吗？

Question

Any option in Kafka Connect to specify from which partition specifically to read the messages. Basically, I am looking for an option in Kafka Connects to manually assign a list of partitions to read.

Similar to assign() method in KafkaConsumer API

https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assign(java.util.Collection)

Answer 1

You can't listen to only specific partition in Kafka Connect.

But you can achieve functionality of inserting messages from only specific partitions.

To have such feature you need to implement your custom Transformation . If Transformation returns null message is skipped, so your custom Transformation must return null for unwanted partitions.

Sample code will be as follow:

public class PartitionFilter <R extends ConnectRecord<R>> implements Transformation<R> {

    public static final ConfigDef CONFIG_DEF = new ConfigDef();

    @Override
    public void configure(Map<String, ?> props) {
        final SimpleConfig config = new SimpleConfig(CONFIG_DEF, props);
    }

    @Override
    public R apply(R record) {
        int neededPartition = 1; // some parititon
        if (record.kafkaPartition() != neededPartition)
           return null;
        return record;
    }

    @Override
    public void close() {
    }

    @Override
    public ConfigDef config() {
        return CONFIG_DEF;
    }
}

More information about transformations can be found: https://kafka.apache.org/documentation/#connect_transforms