I see errors using node-rdkafka but it seems to be working ok

Question

I have a Bluemix Node.js (6.1.0) application that uses node-rdkafka 1.0.3. It seems to be working ok but there are tons of error events like Error: Local: Broker Transport Failure or Error: Local: Authentication failure . The producer options I have set are:

var producer_opts = {
  "metadata.broker.list":env.messagehub.brokers,
  "security.protocol":"sasl_ssl",
  "ssl.ca.location":env.messagehub.calocation,
  "sasl.mechanisms":"PLAIN",
  "sasl.username":env.messagehub.user,
  "sasl.password":env.messagehub.password,
  "api.version.request":true,
  "socket.timeout.ms": 10000,
  "dr_msg_cb":true
};

Consumer has similar settings plus the group.id tag.

I wonder if I should be worrying for theese errors and if there is a way to eliminate them. Thanks!

Answer 1

You are probably hitting https://github.com/edenhill/librdkafka/issues/1218 .

In many cases, as you've noticed, these errors are harmless. The library node-rdkafka is based onto, librdkafka, always connects to all brokers in the cluster. Brokers your applications doesn't interact with will close the idle connections after a while leading to these error messages in your clients.

Unfortunately we don't have a recommended way to eliminate them at the moment. We are currently working on a potential solution to at least reduce their rate and maybe get rid of them.

Update:

With the most recent releases of node-rdkafka (>2.2), you can get rid of all the noisy logs by setting the following properties when creating clients:

'broker.version.fallback': '0.10.2.1',
'log.connection.close' : false