Create Kafka topic with N partitions

Question

I'm using Confluent Kafka DotNet lib to create and produce to topics:

producer.ProduceAsync(topic, key, message);

When this topic doesn't exist, the topic is automatically created with one partition.

But for testing purposes, I want to be able to create a topic with N partitions programmatically.

I'm unable to find any examples here (branch 0.11.5), how can I create a topic with multiple partitions? https://github.com/confluentinc/confluent-kafka-dotnet

Answer 1

Since I was already using Docker I found out it was easiest to simply create a test topic (with N partitions) in the docker-compose:

  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "PartitionsTest:3:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    links:
      - zookeeper

Which creates topic "PartitionsTest" with 3 partitions.

Warning: Your tests may execute before the topic has time to create itself.

Once the Admin feature is available in a release version of the Confluent Kafka library, I'll probably update it to use that approach.