kafka nodejs-使用SASL身份验证创建生产者和使用者的问题

Question

I am trying to create Kafka producer and consumer with SASL authentication using nodejs but it seems to be not available in any of the nodejs kafka packages which I have tried almost all of them (node-rdkafka, kafka-node, no-kafka....)

Tried below option with node-rdkafka but no luck in making publishing msgs with sasl

var Kafka = require('node-rdkafka');
var producer = Kafka.Producer({
  'debug': 'all',
  'metadata.broker.list': 'localhost:9092',
  'security.protocol': 'sasl_plaintext',
  'sasl.username': 'root',
  'sasl.password': 'admin!',
  'sasl.mechanisms': 'PLAIN',
});

// producer.connect();
producer.connect(null, (err, metadata) => {
   console.log(metadata);
   console.error(err);
   console.log('Connected')
});

producer.on('ready', function () {
  try {
    producer.produce('topic1', null, new Buffer('Awesome'), null, Date.now())
  } catch (err) {
    console.log('A error occured')
  }
});

// Any errors we encounter, including connection errors
producer.on('event.error', function(err) {
  console.log('Error from producer');
  console.log(err);
})

producer
  .on('event.log', function(event) {
      console.log(event)
      const loggedEvent = {
        severity: event.severity,
        fac: event.fac
      };

      if (event.severity >= 7) {
        console.log(loggedEvent, event.message);
      } else if (event.severity === 6 || event.severity === 5) {
        console.log(loggedEvent, event.message);
      } else if (event.severity === 4) {
        console.log(loggedEvent, event.message);
      } else if (event.severity > 0) {
        console.log(loggedEvent, event.message);
      } else {
        console.log(loggedEvent, event.message);
      }
})

On top of that facing issues with node-gyp and docker issues due to C++ wrapper for node-rdkafka

https://github.com/nodejs/node/issues/17732
https://github.com/GaiamTV/kafka-node-topic-consumer/issues/3
https://github.com/Blizzard/node-rdkafka/issues/323

Even created jar file with producer and consumer standalone java classes and tried to run those java classes using nodejs child process which didnt throw any errors and at the sametime no output

var cmd    = require('child_process').spawn('java', ['-cp', 'NodeKafka-0.0.1-SNAPSHOT.jar', 'node/kafka/NodeKafka/nodeKafkaConsumer.class']);

//console.log(spawn);
setInterval(function() {
cmd.stdout.on('data', function (data) {
  console.log('stdout: ' + data); // This will print string returned by Main class.
});

},3000);

Lot of dependencies and errors for Kafka nodejs compared to Java and python

Could you please help in using node-rdkafka or any other pacakge that supports SASL authentication(without using SSL) for creating kafka producer and consumer

Answer 1

If you want to use Nodejs with Kafka and SASL you don't have many options. As far as I know, only node-rdkafka officially supports it. That said, I've been using it for over a year now (with SASL) and it's a pretty good client.

I don't know your configuration or use case but note that when using SASL Plain over a plaintext connection, credentials will be sent over the network in plaintext.

When installing node-rdkafka you need to make sure it successfully builds and has all the required features enabled (SASL). Depending on your OS the exact dependencies you need to have installed vary slightly.

Looking at the log you pasted in the node-rdkafka issue, you are missing all dependencies including a C compiler ! The required libraries are listed on https://github.com/edenhill/librdkafka#requirements

Regarding the SASL dependencies, in the npm output, you want to see:

checking for libsasl2 (by pkg-config)... ok
checking for libsasl2 (by compile)... ok (cached)
...
ENABLE_SASL              y
...
LIBS                     ... -lsasl2 ...

Finally your Nodejs client logic looks fine, so once you get your dependencies sorted, it should work.

Answer 2

For kafka with node their is good package https://kafka.js.org/

they have SASL https://kafka.js.org/docs/configuration#a-name-sasl-a-sasl