kafka failing to start - too many open files

Question

I am trying to start the Kafka service but it is giving the below error.

● kafka.service - Apache Kafka server (broker)    Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)   Active: failed (Result: exit-code) since Wed 2021-11-17 06:16:12 UTC; 46min ago
     Docs: http://kafka.apache.org/documentation.html   Process: 10979 ExecStop=/opt/deployments/commoninfra/kafka/bin/kafka-server-stop.sh (code=exited, status=1/FAILURE)   Process: 10409 ExecStart=/opt/deployments/commoninfra/kafka/bin/kafka-server-start.sh /opt/deployments/commoninfra/kafka/config/server.properties  Main PID: 10409 (code=exited, status=1/FAILURE)

Nov 17 06:16:11 atl-kafka2 kafka-server-start.sh[10409]:         at kafka.network.Acceptor.accept(SocketServer.scala:642) Nov 17 06:16:11 atl-kafka2 kafka-server-start.sh[10409]:         at kafka.network.Acceptor.run(SocketServer.scala:571) Nov 17 06:16:11 atl-kafka2 kafka-server-start.sh[10409]:         at java.lang.Thread.run(Thread.java:748) Nov 17 06:16:11 atl-kafka2 kafka-server-start.sh[10409]: [2021-11-17 06:16:11,519] ERROR Error while accepting connection (kafka.network.Acceptor) Nov 17 06:16:11 atl-kafka2 kafka-server-start.sh[10409]: java.io.IOException: Too many open files Nov 17 06:16:12 atl-kafka2 systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE Nov 17 06:16:12 atl-kafka2 kafka-server-stop.sh[10979]: No kafka server to stop Nov 17 06:16:12 atl-kafka2 systemd[1]: kafka.service: Control process exited, code=exited status=1 Nov 17 06:16:12 atl-kafka2 systemd[1]: kafka.service: Unit entered failed state. Nov 17 06:16:12 atl-kafka2 systemd[1]: kafka.service: Failed with result 'exit-code'. 

How to check the below points Find out from which microservice there are many connections to Kafka A total number of established and waiting connections.

we tried to quick-fix by increasing ulimit but we are facing this issue every day. Need permanent solution

Answer 1

To debug the issue, try issuing lsof on the Kafka broker process to get the list of files that kafka broker has opened. Topic partitions are folders.

Check your segment.ms or segment.bytes to see if Kafka is set to rollover new segments frequently which is ending up in creating a lot of files.

You may also want to consider adding new brokers if needed .

Ensure that you are using Kafka 2.1.1+. Check out this ticket https://issues.apache.org/jira/browse/KAFKA-7697