UDP datagram sockets - What are the possible causes of failure of send() for “No buffer space available”

Question

I am testing my code on a cheap Digital Ocean droplet (2GB ram, but I also tried adding 4GB of swap and it didn't change anything).

A part of my program has many UDP client DGRAM sockets open to various local ports (from/to 127.0.0.1).

Sometimes, without an insane amount of data being sent, the send() call fails with No buffer space available (error 105).

My program then tries to use all other sockets available (to other local ports) but they all fail at the same time.

I have set /proc/sys/net/core/wmem_max (and /proc/sys/net/core/rmem_max ) to 16MB, and each of these sockets has setsockopt SOL_SOCKET, SO_SNDBUF set to 8MB. None of these change anything.

How can send() fail with No buffer space available ? Isn't this supposed to only happen when we send() too quickly? But here this is local and I don't send a lot of data.

What puzzles me the most is that when the process retries any another socket it gets the same error. Aren't sockets buffers supposed to be completely independent?

What are the possible causes?

Answer 1

The error No buffer space available does not mean the socket buffer is full. It means that linux kernel cannot allocate a memory for the socket buffer. There is no memory for any socket buffer so when this condition occurs it is not possible to send any data over any socket till some memory is freed. Increasing wmem_max and rmem_max may make the problem even worse because they may increase memory consumption per socket. You can check the overall memory consumption and how much memory is allocated for udp socket buffers:

$ cat /proc/net/sockstat
sockets: used 315
TCP: inuse 8 orphan 0 tw 0 alloc 13 mem 1
UDP: inuse 3853 mem 240812
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

UDP sockets in this example use 240812 pages per 4KB and that is ~940MB. There are 3853 opened sockets

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.8G         75M        1.2M         66M         28M
Swap:            0B          0B          0B

The system i this example has 2 GB physical memory and very low free memory. It is highly probable that a send function fails with the error No buffer space available .

If the memory is actually consumed by socket buffers then adding swap hardly helps because AFAIK the buffers cannot be swapped out.

You can try a VM with more memory. It may help or at least postpone the problem.

You can also audit the application and check how are sockets used. High memory consumption by UDP socket buffers may be caused by an error in the application or by a wrong design. For example if an application listens on UDP socket and there are data but application does not read anything then kernel does not free the memory till the process terminates or till application calls recv . A lot of opened sockets without calling recv may use up memory.