python的asyncio模塊的最大連接並發性

Question

設置-生產者在主機A上運行。創建了數千個協程，每個協程都嘗試向主機B上的使用者發送消息。

import asyncio

NUM_MSGS = 15000
CONSUMER_IP = "<Host B's IP>"
CONSUMER_PORT = <Port #>


@asyncio.coroutine
def send_msg(i, loop):
    conn = asyncio.open_connection(host=CONSUMER_IP, port=CONSUMER_PORT, loop=loop)
    reader, writer = yield from conn

    writer.write(bytes(i, 'utf-8'))

    response = yield from reader.read(100)

    print('{0} - {1}'.format(i, response))
    writer.close()

@asyncio.coroutine
def msg_controller(loop):
    conns = []
    for i in range(NUM_MSGS):
        conns.append(asyncio.async(send_msg(str(i), loop)))
    yield from asyncio.wait(conns)

loop = asyncio.get_event_loop()
loop.run_until_complete(msg_controller(loop))
loop.close()

主機B上的使用者僅接受連接並對其進行響應。

import asyncio

L_PORT = <Port #>
L_HOST = "Host B's IP"
LOAD = 0 #seconds


def handle_connection(transport, data):
    print('Processed {0}'.format('.'))
    transport.write(pickle.dumps('Done'))

class ConsumerProtocol(asyncio.Protocol):

    def connection_made(self, transport):
        self.transport = transport

    def connection_lost(self, e):
        if e:
            print(e)

    def data_received(self, data):
        loop.call_later(LOAD, handle_connection, self.transport, data)


loop = asyncio.get_event_loop()

coro = loop.create_server(ConsumerProtocol, L_HOST, L_PORT)
server = loop.run_until_complete(coro)
print('Server running at {0} on port {1}...'.format(L_HOST, L_PORT))

try:
    loop.run_forever()
except KeyboardInterrupt:
    print('Server stopped')

server.close()
loop.run_until_complete(server.wait_closed())
loop.close()

現在，當NUM_MSGS <= 5000時，設置工作正常。超過5000（大約）協程，例如15000，在生產者端我看到數百個以下錯誤-

Traceback (most recent call last):
  File "/usr/lib64/python3.4/asyncio/tasks.py", line 234, in _step
    result = coro.throw(exc)
  File "aggressive_producer.py", line 15, in send_msg
    response = yield from reader.read(100)
  File "/usr/lib64/python3.4/asyncio/streams.py", line 452, in read
    yield from self._wait_for_data('read')
  File "/usr/lib64/python3.4/asyncio/streams.py", line 393, in _wait_for_data
    yield from self._waiter
  File "/usr/lib64/python3.4/asyncio/futures.py", line 386, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib64/python3.4/asyncio/tasks.py", line 287, in _wakeup
    value = future.result()
  File "/usr/lib64/python3.4/asyncio/futures.py", line 275, in result
    raise self._exception
  File "/usr/lib64/python3.4/asyncio/selector_events.py", line 662, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
Task exception was never retrieved

（請忽略我沒有處理異常的事實）

令人驚訝的是，在消費者方面，沒有看到錯誤（connection_lost（）沒有報告任何異常）。 實際上，消費者甚至看不到在生產者端看到上述錯誤的連接（未收到connection_made（）cb）。 但是，上述異常似乎是在等待消費者響應時發生的。

我檢查了日志（/ var / log / messages），但也沒有發現任何錯誤。

這是異步限制嗎？ 異步中的最大並發性有這樣的限制嗎？

操作系統：RHEL 6.5 | python版本：3.4.3

PS-我增加了最大開放fd限制，最大tcp syn積壓，可用端口范圍和啟用了tcp_tw_reuse。

Answer 1

您肯定沒有資源。

Connection reset by peer消息Connection reset by peer告訴該問題不是在客戶端，而是在服務器上。 我猜連接被服務器的防火牆或反垃圾郵件系統拒絕了。

Answer 2

$ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-m: resident set size (kbytes)      unlimited
-u: processes                       30197
-n: file descriptors                1024
-l: locked-in-memory size (kbytes)  64
-v: address space (kbytes)          unlimited
-x: file locks                      unlimited
-i: pending signals                 30197
-q: bytes in POSIX msg queues       819200
-e: max nice                        0
-r: max rt priority                 0
-N 15:                              unlimited

$ ulimit -n 15000