使用带有 https 的 cherrypy 时加载大文件时抛出异常的原因是什么？

Question

System Info

$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

$ uname -a
Linux lb-cam-11 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ python --version
Python 3.7.1

$ openssl version
OpenSSL 1.0.1h 5 Jun 2014

>>> cherrypy.__version__
'18.1.0'

>>> import ssl
>>> [(ele, sys.modules[ele]) for ele in sys.modules if 'ssl' in ele]
[('ssl',
  <module 'ssl' from '/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/ssl.py'>),
 ('_ssl',
  <module '_ssl' from '/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so'>)]
>>>

>>> cheroot.__version__
'6.5.4'

---

Hi

I am attempting to set up a cherrypy server that uses https.

The ssl related cherrypy configuration I am using looks like:

server_config={
    'server.socket_host': chrysocketserver,
    'server.socket_port':int(chrysocketport),
    'server.ssl_module':'pyopenssl',
    'server.ssl_certificate':sslcertificate,
    'server.ssl_private_key':privatekey,
    'server.ssl_certificate_chain':certificatechain,
}
cherrypy.config.update(server_config)

With small pages this looks like it works fine, but when I try to load a large page (I have not yet worked out at what point the problem starts occurring) the cheroot back-end (I know nothing about this) throws an exception that looks like:

Traceback (most recent call last):
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/server.py", line 1252, in communicate
    req.respond()
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/server.py", line 1056, in respond
    self.server.gateway(self).respond()
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/wsgi.py", line 147, in respond
    self.write(chunk)
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/wsgi.py", line 228, in write
    self.req.write(chunk)
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/server.py", line 1112, in write
    self.conn.wfile.write(chunk)
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/makefile.py", line 36, in write
    self._flush_unlocked()
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
    n = self.raw.write(bytes(self._write_buf))
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/socket.py", line 607, in write
    return self._sock.send(b)
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1729, in send
    self._raise_ssl_error(self._ssl, result)
  File "/tools/oss/packages/x86_64-centos7/python/3.7.1/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1616, in _raise_ssl_error
    raise WantWriteError()
OpenSSL.SSL.WantWriteError

(Some of the page is loaded, but not all)

Has anyone seen this issue before? and know what might be causing it? On the surface it looks like the socket being written to becomes unwritable (because the buffer is full?) and when it encounters the SSL.WantWriteError it simply raises an exception (rather than waiting and trying again). But I am probably being naive in my reading of it.

Does anyone know I how I might get round this problem?

You're incorrectly assuming that the stdlib ssl

Yes post-posting I noticed this and replied as an Answer (repeated below) which got deleted (probably because it wasn't actually an answer) - the actual version used is 1.0.2m:

Apologies - the ssl version I posted was simply the output of which openssl, however it the python is using a later version (1.0.2m):

>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2m  2 Nov 2017'

Is that also too old? (I believe it supports TLS 1.2, but I do not know what cyphers it supports or otherwise)

I did think it unlikely to be an ssl issue since small pages are loaded fine.

Either way in this environment my hands are tied regarding the ssl version being used. In order to test I need to build python + openssl (plus any other dependencies) on a local machine, so it may be a while before I find time to test it.

Answer 1

Apologies - the ssl version I posted was simply the output of which openssl, however it the python is using a later version (1.0.2m):

>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2m  2 Nov 2017'

Is that also too old? (I believe it supports TLS 1.2, but I do not know what cyphers it supports or otherwise)

I did think it unlikely to be an ssl issue since small pages are loaded fine.

Either way in this environment my hands are tied regarding the ssl version being used. In order to test I need to build python + openssl (plus any other dependencies) on a local machine, so it may be a while before I find time to test it.

Answer 2

To wrap up and give an answer to my question.

There appears to be a bug in pyopenssl ssl_module related to the way OpenSSL.SSL.WantReadError and OpenSSL.SSL.WantWriteError are handled as documented here: https://github.com/cherrypy/cheroot/issues/245

If I configure cherrypy using the 'builtin' module as below:

server_config={
    'server.socket_host': chrysocketserver,
    'server.socket_port':int(chrysocketport),
    'server.ssl_module':'builtin',
    'server.ssl_certificate':sslcertificate,
    'server.ssl_private_key':privatekey,
    'server.ssl_certificate_chain':certificatechain,
}
cherrypy.config.update(server_config)

it works fine (at least in my environment)