如何摆脱“使用 timeoutMillis = ... 关闭 Kafka 生产者”

Question

I am sending Apache Avro formatted messages to a Kafka broker instance via the following code:

        ProducerRecord<String, byte[]> producerRecord = new ProducerRecord<>(kafkaTopic.getTopicName(), null, null,
                avroConverter.getSchemaId().toString(), convertRecordToByteArray(kafkaRecordToSend));

        String avroSchemaName = null;

        // some of my AVRO schemas are unions, some are simple:
        if (_avroSchema.getTypes().size() == 1) {
            avroSchemaName = _avroSchema.getTypes().get(0).getName();
        } else if (_avroSchema.getTypes().size() == 2) {
            avroSchemaName = _avroSchema.getTypes().get(1).getName();
        }

        // some custom header items...
        producerRecord.headers().add(MessageHeaders.MESSAGE_ID.getText(), messageID.getBytes());
        producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SUBJECT.getText(),
                avroSchemaName.getBytes());
        producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SCHEMA_ID.getText(),
                avroConverter.getSchemaId().toString().getBytes());
        if (multiline) {
            producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_MULTILINE_RECORD_NAME.getText(),
                    MULTILINE_RECORD_NAME.getBytes());
        }

        try {
            Future<RecordMetadata> result = kafkaProducer.send(producerRecord);
            RecordMetadata sendResult = result.get();
            MessageLogger.logResourceBundleMessage(_messages, "JAPCTOAVROKAFKAPRODUCER:DEBUG0002",
                    sendResult.offset());
        } catch (Exception e) {
            MessageLogger.logError(e);
            throw e;
        }

The code works fine, the messages end up in Kafka and are processed to end up in an InfluxDB. The problem is that every send operation produces a lot of INFO messages (client ID number is an example):

• [Producer clientId=producer-27902] Closing the Kafka producer with timeoutMillis = 10000 ms.
• [Producer clientId=producer-27902] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
• Kafka startTimeMs: ...
• Kafka commitId: ...
• [Producer clientId=producer-27902] Cluster ID:

which Spam our Graylog.

I use similar code to send String formatted messages. This code is executed without producing INFO messages...

ProducerRecord<String, String> recordToSend = new ProducerRecord<>(queueName, messageText);
    recordToSend.headers().add("messageID", messageID.getBytes());

    Future<RecordMetadata> result = _producerConnection.send(recordToSend);
    

I know that the INFO message are logged from class org.apache.kafka.clients.producer.KafkaProducer . I need to get rid of these messages, but I do not have access to the logging.mxl defining the logger properties for Graylog.

Is there a way to get rid of these messages via POM-entries or programatically?

Answer 1

The reason of the code behavior was a design flaw: The code in the post above was placed in a method which was called for sending the message to Kafka. The KafkaProducer class was instantiated in that method and each time when the method was called. Surprisingly, KafkaProducer issues the Closing the KafkaProducer with timeoutMillis = not only at an explicit close() by the calling code, but as well when the strong reference to the instance is lost ( in my case when the code leaves the method), In the latter case, the timeoutMillis is set to 9223372036854775807 (largest long number).

To get rid of the many messages, I moved the KafkaProducer instantiation out of the method and made the instance variable a class attribute and I do not call an explicit close() after the send(...) any more.

Furthermore I changed the instance of my class instantiating KafkaProducer to a strong referencing class member.

By doing so, I get some messages by the KafkaProducer at instantiation, then there is silence.