SSL 传输上的 Asyncio 致命错误 - IndexError Deque Index Out Of Range

Question

I keep running into the bellow error and can't work out how to fix it or whether it is a bug in asyncio that I need to report. My program works fine is it basically the same as the example in there docs but it can send lots of messages upto 20 per second to multiple clients when there is a low number of updates it works fine but when the number of writes increases I run into this error.

I am using create_server from asyncio to make the server listen for new clients and I believe this creates a new connection on each connection. Not all clients stop only one clients stops and the rest keep running it's usually the last client that connected but not always. I have played with the asyncio/sslproto.py file removing the reference to [0] and replacing them with popleft which they got me the not very helpful SSL: BAD_LENGTH error which I have had before and using asyncio was supposed to be the fix.

When I logged the value of self._write_backlog it returned the deque as expected and before the error there was clearly data in it and I believe that self._write_backlog[0] should return the leftmost element in the deque which there appeared to be one.

Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f267462e780>
transport: <_SelectorSocketTransport fd=38 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
    File "/usr/local/lib/python3.7/asyncio/sslproto.py", line 689, in 
    _process_write_backlog
  del self._write_backlog[0]
IndexError: deque index out of range

On further investigation the same error also seems to happen at a diffrent point in the code but it is the same error like it's trying to access an empty deque.

Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f45f802ec88>
transport: <_SelectorSocketTransport fd=29 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
    File "/usr/local/lib/python3.7/asyncio/sslproto.py", line 664, in _process_write_backlog
    data, offset = self._write_backlog[0]
IndexError: deque index out of range

EDIT

The version of python I am running on is 3.7.1 but I have also tried 3.7.3 which gave the same error.

UPDATE

It is believed that this issue is caused by an incompatibility between asyncio and OpenSSL https://bugs.python.org/issue37226

UPDATE 2

A minimal example is available on GitHub

Answer 1

The problem here is related to system resources when you connect lots of clients it can cause CPU or Memory to become exhausted which then causes this error as the write fails.

The solution is to implement flow control to limit the amount of data being sent at one time and stop resources becoming exhausted the following article has a good example of simple flow control that blocks while old data is removed then resumes using the high and low water marks. When the data exceeds the high water mark it triggers pause_writing then when the data size reduces to lower than the low water mark writing resumes this can cause a noticeable slow down dependent on load.

As an alternative to adding your own flow control to a protocol the asyncio Streams primitives offer a simple way to write and read data that includes flow control by default but for me made implementation harder so I opted to include my own at the protocol level as shown in the linked article.