ArangoDB Could not connect

Question

arangod is running for some time without any problems, but at some point no more connections can be made. aranogsh then shows the following error message:

Error message 'Could not connect to 'tcp://127.0.0.1:8529' 'connect() failed with #99 - Cannot assign requested address''

In the log file arangod still writes more trace information.

After restarting aranogd it is running without problems again, until the problem suddenly reoccurs.

Why is this happening?

Answer 1

Since this question was sort of answered by time, I'll use this answer to elaborate howto dig into such a situation and to get a valuable analysis on which operating system parameters to look. I'll base this on linux targets.

First we need to find out whats currently going on using the netstat tool as a root user (we care for tcp ports only):

netstat -alnpt
Proto Recv-Q Send-Q Local Address     Foreign Address    State       PID/Program name
...
tcp        0      0 0.0.0.0:8529      0.0.0.0:*          LISTEN      3478/arangod
tcp        0      0 127.0.0.1:45218   127.0.0.1:8529     ESTABLISHED 6902/arangosh
tcp        1      0 127.0.0.1:46985   127.0.0.1:8529     CLOSE_WAIT  485/arangosh

We see an overview of the 3 possible value groups:

• LISTEN : These are daemon processes offering tcp services to remote ends, in this case the arangod process with its server socket. It binds port 8529 on all available ipv4 addresses of the system ( 0.0.0.0 ) and accepts connections from any remote location ( 0.0.0.0:* )
• ESTABLISHED : this is an active tcp connection in this case between arangosh and arangod ; Arangosh has its client port ( 45218 ) in the higher range connecting arangod on port 8529 .
• CLOSE_WAIT : this is a connection in termination state. Its normal to have them. The TCP stack of the operating system keeps them around for a while to have a knowledge where to sort in stray TCP-packages that may have been sent, but did not arive on time.

As you see TCP ports are 16 bits unsigned integers, ranging from 0 to 65535 . Server sockets start from the lower end, and most operating systems require processes to be running as root to bind ports below 1024. Client sockets start from the upper end and range down to a specified limit on the client. Since multiple clients can connect one server, while the server port range seems narrow, its usually the client side ports that wear out. If the client frequently closes and reopens the connection you may see many sockets in CLOSE_WAIT state, as many discussions across the net hint, these are the symptoms of your system eventually running out of resources. In general the solution to this problem is to to re-use existing connections through the keepalive feature.

As the solaris ndd command explains thoroughly which parameters it may modify with which consequences in the solaris kernel, the terms explained there are rather generic to tcp sockets, and may be found on many other operating systems in other ways; in linux - which we focus on here - through the /proc/sys/net -filesystem.

Some valuable switches there are:

• ipv4/ip_local_port_range This is the range for the local sockets. You can try to narrow it, and use arangob --keep-alive false to explore whats happening if your system runs out of these.
• time wait (often shorted to tw ) is the section that controls what the TCP-Stack should do with already closed sockets in CLOSE_WAIT state. The Linux kernel can do a trick here - it can instantly re-use connections in that state for new connections. Vincent Bernat explains very nicely which screws to turn and what the differnt parameters in the kernel mean.

So once you decided to change some of your values in /proc so your host better scales to the given situation, you need to make them reboot persistant - since /proc is volatile and won't remember values across reboots.

Most linux systems therefore offer the /etc/sysctl.[d|conf] file; It maps slashes in the proc filesystem to dots, so /proc/sys/net/ipv4/tcp_tw_reuse will translate into net.ipv4.tcp_tw_reuse .