如何在nodejs中创建带分区的kafka主题？

Question

I am using kafka-node link api for creating kafka topics. I did not find how to create a kafka topic with partitions.

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    client = new kafka.Client(),
    producer = new Producer(client);
// Create topics sync
producer.createTopics(['t','t1'], false, function (err, data) {
    console.log(data);
});
// Create topics async
producer.createTopics(['t'], true, function (err, data) {});

producer.createTopics(['t'], function (err, data) {});// Simply omit 2nd arg

how to create kafka topic with partitions in nodejs.

Answer 1

From your node.js app execute the shell script $KAFKA_HOME/bin/kafka-topics.sh —create —topic topicname —partitions 8 —replication-factor 1 —zookeeper localhost:2181

Where $KAFKA_HOME is the location where you installed Kafka

Answer 2

As documentation describes, this method works only when auto.create.topics.enable is set to true :

This method is used to create topics on the Kafka server. It only works when auto.create.topics.enable, on the Kafka server, is set to true. Our client simply sends a metadata request to the server which will auto create topics. When async is set to false, this method does not return until all topics are created, otherwise it returns immediately.

This means that any operation on unknown topic will lead to its creation with default number of partitions configured by num.partitions parameter.

I'm not sure, but maybe one of the node-rdkafka implementations could allow you to call corresponding librdkafka method to create topic?

Answer 3

I am not that sure but I think as per your requirement the code has updated here:- https://github.com/SOHU-Co/kafka-node#createtopicstopics-cb , adding a parameter "replicaAssignment".

  // Optional explicit partition / replica assignment
  // When this property exists, partitions and replicationFactor properties are ignored
  replicaAssignment: [
    {
      partition: 0,
      replicas: [3, 4]
    },
    {
      partition: 1,
      replicas: [2, 1]
    }
  ]

Answer 4

The Producer.createTopics takes a partitons option. See https://www.npmjs.com/package/kafka-node#createtopicstopics-cb

Pass an object, rather than a string

producer.createTopics(['t', 't1'], true, function (err, data) {});

becomes

producer.createTopics(
  [
    { topic: 't', paritions: 5 },
    { topic: 't1', partitions: 23 },
  ], 
  true,
  function (err, data) {}
);