I am sending Apache Avro formatted messages to a Kafka broker instance via the following code:
ProducerRecord<String, byte[]> producerRecord = new ProducerRecord<>(kafkaTopic.getTopicName(), null, null,
avroConverter.getSchemaId().toString(), convertRecordToByteArray(kafkaRecordToSend));
String avroSchemaName = null;
// some of my AVRO schemas are unions, some are simple:
if (_avroSchema.getTypes().size() == 1) {
avroSchemaName = _avroSchema.getTypes().get(0).getName();
} else if (_avroSchema.getTypes().size() == 2) {
avroSchemaName = _avroSchema.getTypes().get(1).getName();
}
// some custom header items...
producerRecord.headers().add(MessageHeaders.MESSAGE_ID.getText(), messageID.getBytes());
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SUBJECT.getText(),
avroSchemaName.getBytes());
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_REGISTRY_SCHEMA_ID.getText(),
avroConverter.getSchemaId().toString().getBytes());
if (multiline) {
producerRecord.headers().add(MessageHeaders.AVRO_SCHEMA_MULTILINE_RECORD_NAME.getText(),
MULTILINE_RECORD_NAME.getBytes());
}
try {
Future<RecordMetadata> result = kafkaProducer.send(producerRecord);
RecordMetadata sendResult = result.get();
MessageLogger.logResourceBundleMessage(_messages, "JAPCTOAVROKAFKAPRODUCER:DEBUG0002",
sendResult.offset());
} catch (Exception e) {
MessageLogger.logError(e);
throw e;
}
The code works fine, the messages end up in Kafka and are processed to end up in an InfluxDB. The problem is that every send operation produces a lot of INFO messages (client ID number is an example):
which Spam our Graylog.
I use similar code to send String formatted messages. This code is executed without producing INFO messages...
ProducerRecord<String, String> recordToSend = new ProducerRecord<>(queueName, messageText);
recordToSend.headers().add("messageID", messageID.getBytes());
Future<RecordMetadata> result = _producerConnection.send(recordToSend);
I know that the INFO message are logged from class org.apache.kafka.clients.producer.KafkaProducer
. I need to get rid of these messages, but I do not have access to the logging.mxl defining the logger properties for Graylog.
Is there a way to get rid of these messages via POM-entries or programatically?
The reason of the code behavior was a design flaw: The code in the post above was placed in a method which was called for sending the message to Kafka. The KafkaProducer
class was instantiated in that method and each time when the method was called. Surprisingly, KafkaProducer
issues the Closing the KafkaProducer with timeoutMillis =
not only at an explicit close() by the calling code, but as well when the strong reference to the instance is lost ( in my case when the code leaves the method), In the latter case, the timeoutMillis
is set to 9223372036854775807 (largest long number).
To get rid of the many messages, I moved the KafkaProducer
instantiation out of the method and made the instance variable a class attribute and I do not call an explicit close()
after the send(...)
any more.
Furthermore I changed the instance of my class instantiating KafkaProducer
to a strong referencing class member.
By doing so, I get some messages by the KafkaProducer
at instantiation, then there is silence.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.