Too many open files on a project using spring-cloud stream and kafka after upgrade to 2.1

Question

After upgrade to spring cloud stream 2.1 on a project using multi binder kafka(2 brokers) and rabbit(1 broker) we are facing too many open files problem.

The number of files opened keeps growing to the limit defined by the os(redhat 7.3).

I use the actuator endpoint to monitor: /actuator/metrics/process.files.open

{
"name": "process.files.open",
"description": "The open file descriptor count",
"baseUnit": "files",
"measurements": [
{
"statistic": "VALUE",
"value": 1686
}

]
}

Using the version 2.0 of spring cloud stream the problem is not observed, the number of open files is stable aroun 80 files. Using the exact following versions:

 +- org.springframework.cloud:spring-cloud-stream-binder-kafka-core:jar:2.0.0.RELEASE:compile
[INFO] |  |  \- org.springframework.integration:spring-integration-kafka:jar:3.0.3.RELEASE:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:1.0.2:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.4:compile
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  \- org.springframework.kafka:spring-kafka:jar:2.1.10.RELEASE:compile

I'm suspecting that the upgrade to kafka-client 2.0.0 is the potential issue.

In a first attempt I wanted to try the spring cloud stream 2.1 with kafka-clients 1.0.2, according to docs it's possible but I'm facing an issue. Here's my maven configuration with exclusion:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kafka</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-kafka</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>1.0.2</version>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.2.5.RELEASE</version>
</dependency>
<dependency>
    <groupId>org.springframework.integration</groupId>
    <artifactId>spring-integration-kafka</artifactId>
    <version>3.1.0.RELEASE</version>
</dependency>

I'm having the following error:

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.Consumer.poll(Ljava/time/Duration;)Lorg/apache/kafka/clients/consumer/ConsumerRecords;
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:741)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:698)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.base/java.lang.Thread.run(Thread.java:834)

Any idea about too many open files and how to diagnose? How can I test with downgraded kafka clients?

Thanks for you help.

Answer 1

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.Consumer.poll(Ljava/time/Duration;)Lorg/apache/kafka/clients/consumer/ConsumerRecords;

spring-kafka 2.2.x needs 2.0.1 kafka-clients or later.