python的asyncio模块的最大连接并发性

Question

Setup - The producer/s run on Host A. Multiple thousands of coroutines are created each of which try to send a message to the consumer on Host B.

import asyncio

NUM_MSGS = 15000
CONSUMER_IP = "<Host B's IP>"
CONSUMER_PORT = <Port #>


@asyncio.coroutine
def send_msg(i, loop):
    conn = asyncio.open_connection(host=CONSUMER_IP, port=CONSUMER_PORT, loop=loop)
    reader, writer = yield from conn

    writer.write(bytes(i, 'utf-8'))

    response = yield from reader.read(100)

    print('{0} - {1}'.format(i, response))
    writer.close()

@asyncio.coroutine
def msg_controller(loop):
    conns = []
    for i in range(NUM_MSGS):
        conns.append(asyncio.async(send_msg(str(i), loop)))
    yield from asyncio.wait(conns)

loop = asyncio.get_event_loop()
loop.run_until_complete(msg_controller(loop))
loop.close()

The consumer on Host B just accepts a connection & responds to it.

import asyncio

L_PORT = <Port #>
L_HOST = "Host B's IP"
LOAD = 0 #seconds


def handle_connection(transport, data):
    print('Processed {0}'.format('.'))
    transport.write(pickle.dumps('Done'))

class ConsumerProtocol(asyncio.Protocol):

    def connection_made(self, transport):
        self.transport = transport

    def connection_lost(self, e):
        if e:
            print(e)

    def data_received(self, data):
        loop.call_later(LOAD, handle_connection, self.transport, data)


loop = asyncio.get_event_loop()

coro = loop.create_server(ConsumerProtocol, L_HOST, L_PORT)
server = loop.run_until_complete(coro)
print('Server running at {0} on port {1}...'.format(L_HOST, L_PORT))

try:
    loop.run_forever()
except KeyboardInterrupt:
    print('Server stopped')

server.close()
loop.run_until_complete(server.wait_closed())
loop.close()

Now, the setup works fine when NUM_MSGS <= 5000. Beyond 5000 (approx.) coroutines, say 15000, I see multiple hundreds of below error on the producer side -

Traceback (most recent call last):
  File "/usr/lib64/python3.4/asyncio/tasks.py", line 234, in _step
    result = coro.throw(exc)
  File "aggressive_producer.py", line 15, in send_msg
    response = yield from reader.read(100)
  File "/usr/lib64/python3.4/asyncio/streams.py", line 452, in read
    yield from self._wait_for_data('read')
  File "/usr/lib64/python3.4/asyncio/streams.py", line 393, in _wait_for_data
    yield from self._waiter
  File "/usr/lib64/python3.4/asyncio/futures.py", line 386, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib64/python3.4/asyncio/tasks.py", line 287, in _wakeup
    value = future.result()
  File "/usr/lib64/python3.4/asyncio/futures.py", line 275, in result
    raise self._exception
  File "/usr/lib64/python3.4/asyncio/selector_events.py", line 662, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
Task exception was never retrieved

(Please ignore the fact that I haven't handled exceptions)

What's surprising is that on the consumer side, no errors are seen (connection_lost() doesn't report any exception). In fact, the consumer does not even see the connections for which the above error is seen on the producer side (connection_made() cb is not received). Yet, the exception above seems to have occurred while waiting for a response from the consumer.

I checked the logs (/var/log/messages) but did not find any errors there either.

Is this an asyncio limitation? Is there such a limit on max concurrency in asyncio?

OS: RHEL 6.5 | python version: 3.4.3

PS - I have increased max open fd limit, max tcp syn backlog, usable port range, and enabled tcp_tw_reuse.

Answer 1

You have definitely run out of resources.

Connection reset by peer message tells that problem not on client side but on server. I guess connections are rejected by server's firewall or antispam system.

Answer 2

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       30197
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  64
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 30197
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

$ ulimit -n 15000