OSError: [Errno 107] 在 Jupyter docker 容器中处理大文件时

Question

I am working with Jupyter notebook (.ipynb) files in a docker container. I have mounted a backup file containing ~16 million entries of SQL, which I want to operate on. The entries in the backup file are of PostgreSQL dialect, and are raw SQL somewhat as follows:

INSERT INTO table_name([column_names separated by comma]) VALUES (values separated by comma);

I am having problems dealing with this file in Jupyter. I tried to read it line by line and insert the lines as raw SQL into PostgreSQL database. This worked for a couple dozen thousand rows but then the OSError: [Errno 107] Transport endpoint is not connected appeared. Then I tried to start writing from the line where I last left off but the error appears soon enough again.

Next, I tried Python's readlines:

with open("/path/to/file.backup", "r") as f:
    content = f.readlines()

but that fails as well, with the same error.

It might be of importance that the backup file is mounted to the docker container as a volume. The container is run using docker-compose and the relevant parts of the docker-compose.yml are here:

version: '2.4'
services:
  volumes:
    - /c/Users/myuser/path/to/notebooks:/tf/notebooks
    - /c/Users/myuser/path/to/backup_file:/tf/data
  ports:
    - 8888:8888
  user: root

The.ipynb file I am working with resides on the same level but in a different directory than the backup file. Both directories are mounted from outside the container.

I am wondering whether this could be like a memory or CPU issue (I did docker stats on my container, lot of CPU usage during handling of the file), but it is still weird to me that it gives this error.

Whenever this error occurs, I am unable to access my notebooks directory in Jupyter, and I have to restart entire Docker (taking the container down and up is not enough).

Relevant logs from Jupyter container: [E 11:48:39.552 LabApp] Error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/filemanager.py", line 471, in save self._save_notebook(os_path, nb) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 293, in _save_notebook with self.atomic_writing(os_path, encoding='utf-8') as f: File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 213, in atomic_writing with atomic_writing(os_path, *args, log=self.log, **kwargs) as f: File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 103, in atomic_writing copy2_safe(path, tmp_path, log=log) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 51, in copy2_safe shutil.copyfile(src, dst) File "/usr/lib/python3.6/shutil.py", line 120, in copyfile with open(src, 'rb') as fsrc: OSError: [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [W 11:48:39.556 LabApp] 500 PUT /api/contents/notebooks/populate_db.ipynb?1591098519530 (172.18.0.1): Unexpected error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [W 11:48:39.556 LabApp] Unexpected error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [E 11:48:39.556 LabApp] { "Host": "localhost:8888", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0", "Accept": "*/*", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "Referer": "http://localhost:8888/lab?", "Content-Type": "application/json", "X-Xsrftoken": "2|695ba9e2|ce49a1a69d34d68ccd56352b0c805223|1591003618", "Origin": "http://localhost:8888", "Content-Length": "6836", "Connection": "keep-alive", "Cookie": "username-localhost-8888=\"2|1:0|10:1590754680|23:username-localhost-8888|44:MDI2OWM5NjVlMjhmNGJjOTgzZjZkNDg3ZDMyNmMyMDc=|f2b616ab71fecbbeac60cfa57455e7d49cdf3563ef00729e1adc3f0c4d17f86e\"; pga4_session=34c2a786-ecfe-48b3-8682-c56b64236b64;3p1/6xx3V2lJFdQouLkwpXmndM8=; _xsrf=2|695ba9e2|ce49a1a69d34d68ccd56352b0c805223|1591003618, PGADMIN_LANGUAGE=en": "Pragma", "no-cache": "Cache-Control": "no-cache" } [E 11:48.39.556 LabApp] 500 PUT /api/contents/notebooks/populate_db?ipynb.1591098519530 (172.18.0.1) 20:92ms referer=http://localhost?8888/lab? [E 11:48:39.552 LabApp] Error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/filemanager.py", line 471, in save self._save_notebook(os_path, nb) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 293, in _save_notebook with self.atomic_writing(os_path, encoding='utf-8') as f: File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 213, in atomic_writing with atomic_writing(os_path, *args, log=self.log, **kwargs) as f: File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 103, in atomic_writing copy2_safe(path, tmp_path, log=log) File "/usr/local/lib/python3.6/dist-packages/notebook/services/contents/fileio.py", line 51, in copy2_safe shutil.copyfile(src, dst) File "/usr/lib/python3.6/shutil.py", line 120, in copyfile with open(src, 'rb') as fsrc: OSError: [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [W 11:48:39.556 LabApp] 500 PUT /api/contents/notebooks/populate_db.ipynb?1591098519530 (172.18.0.1): Unexpected error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [W 11:48:39.556 LabApp] Unexpected error while saving file: notebooks/populate_db.ipynb [Errno 107] Transport endpoint is not connected: '/tf/notebooks/populate_db.ipynb' [E 11:48:39.556 LabApp] { "Host": "localhost:8888", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0", "Accept": "*/*", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "Referer": "http://localhost:8888/lab?", "Content-Type": "application/json", "X-Xsrftoken": "2|695ba9e2|ce49a1a69d34d68ccd56352b0c805223|1591003618", "Origin": "http://localhost:8888", "Content-Length": "6836", "Connection": "keep-alive", "Cookie": "username-localhost-8888=\"2|1:0|10:1590754680|23:username-localhost-8888|44:MDI2OWM5NjVlMjhmNGJjOTgzZjZkNDg3ZDMyNmMyMDc=|f2b616ab71fecbbeac60cfa57455e7d49cdf3563ef00729e1adc3f0c4d17f86e\"; pga4_session=34c2a786-ecfe-48b3-8682-c56b64236b64;3p1/6xx3V2lJFdQouLkwpXmndM8=; _xsrf=2|695ba9e2|ce49a1a69d34d68ccd56352b0c805223|1591003618, PGADMIN_LANGUAGE=en": "Pragma", "no-cache": "Cache-Control": "no-cache" } [E 11:48.39.556 LabApp] 500 PUT /api/contents/notebooks/populate_db?ipynb.1591098519530 (172.18.0.1) 20:92ms referer=http://localhost?8888/lab?

I have not experienced this before, and seem unable to solve. Thanks to anyone who can help!

Answer 1

Since no one took this up, I decided to post my own non-conclusive answer.

The problem (socket error) seems to occur when too much I/O operations occur over a websocket. Since a docker container communicates with docker volumes via websockets, there is a limit to the number of operations per unit of time that can take place.

In my case there was a lot of reading from a volume and simultaneously a lot of writing to a volume.

Possible solutions include:

1. limiting operation amount per unit of time
2. building the volumes into the docker image that is being run, if that is possible

Option 1 could be too slow for some applications, and option 2 might not always be possible due to the changing contents of the volumes.