Go program running out of file handles when there are plenty file handles available

Question

I have a golang program that connects to other services (and a local consul agent) and when talking to the consul agent it's failing with the following error:

Put http://localhost:9501/v1/kv/health_checks/item: dial tcp: lookup localhost: too many open files

And this is all good, I can just check how many open files the system and the program has, right?

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7871
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7871
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

And lsof :

lsof | wc -l
5343

This is still way below the limit. Other programs in the system are all good, I can CURL services, I can CURL this consul agent, create files and everything, but the golang program continues to fail.

Any ideas on how I could debug this? Where should I be looking at?

Answer 1

Any ideas on how I could debug this? Where should I be looking at?

Send the program a SIGQUIT while it is running. This will make a backtrace of all the running go routines. You'll probably find your culprit in there.

My money would be on forgetting to Close() the Body of an http.Response as that is what I do the most often!

Answer 2

如果要连接到不同的端点/服务（即，如果您正在运行搜寻器），请考虑关闭空闲连接。