Kafka server node goes down as with “Too many open files” error

Question

We are using 3 node Kafka cluster, with a total of 151 topics with 1 partition for each topic. And we have configured the replication factor=3. While we start kafka brokers getting following error:

ERROR Error while accepting connection (kafka.network.Acceptor)

java.io.IOException: Too many open files

Answer 1

The default value of max. open files is 1024 on most Unix systems. Depending on your throughput you need to configure a much higher value. Try to start with 32768 or higher

Answer 2

Looks like this is due to the less number of file handles.

Can you check the file descriptor limit as below

ulimit -n

Try changing the open file descriptor to a higher value:

ulimit -n <noOfFiles>

You can get the maximum allowed number of open files: cat /proc/sys/fs/file-max