Strange behaviour in Python SocketServer

Question

I have created a python socket server, using a class inherited from SocketServer.BaseRequestHandler , overriding setup and handle methods. Of cource, SocketServer.BaseRequestHandler.setup is called at the end of my own setup .

This is my server class

class MyServer(SocketServer.ForkingMixIn, SocketServer.TCPServer):
    timeout = 30

A typical forking socket server.

Here is how I run my server

while True:
        try:
            server = MyServer((host, port), MyRequestHandler)
            print('Server listening on', (host, port))
            server.timeout = 300  # seconds
            server.serve_forever()
        except:
            print('Error with server, retrying in 5 seconds...')
            print(sys.exc_info())
            sleep(5)

host and port are predefined, no problem with them.

Server works fine, except when clients count reaches 40. After this number, no new connections will be accepted, all will be refused. I checked this with a client test python script from my own system. Only 40!

Why 40 ? I have checked source code for SocketServer and found nothing related to this. I currently have no clue regarding this issue. Any, and I really mean it, any help is appreciated :))

Thanks in advance

OS: CentOS 6.5

Answer 1

This is probably unrelated to Python. Tune your Linux kernel, in testing phase do stuff like:

• turn syncookies off
• increase file handles available for the user (every socket opened is also a file handle used - maybe you're running out of them?)
• look at stuff like this: http://people.redhat.com/alikins/system_tuning.html#tcp
• and: http://people.redhat.com/alikins/system_tuning.html#fds
• check if stuff like fail2ban is installed ( http://www.fail2ban.org/wiki/index.php/Main_Page )
• check if rate limits are applied by iptables (in testing phase you could do iptables -F after making sure that default chain policy is ACCEPT )
• and last but not in the very least, check dmesg , /var/log/messages , /var/log/syslog , etc

One thing that theoretically might be related to Python is SO_REUSEADDR :

http://www.unixguide.net/network/socketfaq/4.5.shtml

Check if you have it set for your socket.

UPDATE:

I just realized that since the 40 connections that your socket server maxes out at is actually pretty low, the simplest option could be running your socket server through systrace , just use -f flag to track forked processes as well. You could eg start socket server, open 35 simultaneous connections, and then connect systrace to a running process and set up 5 more connections and see what systrace reports. Very often in such situations syscalls fail with errors that are visible in systrace and allow pinpointing root cause relatively easily.

Answer 2

I really have now idea how I missed this in source!

class ForkingMixIn:

    """Mix-in class to handle each request in a new process."""

    timeout = 300
    active_children = None
    max_children = 40

Yeah, now I see the max_children property.

Thanks guys