No space left on device error in Docker when running a Selenium scraper

Question

I am running a relatively straightforward scraper inside a Docker container with compose on my local machine (macOS Mojave). I am using Selenium for scraping. All works fine for a couple of hours, but at some point the container exits and when I inspect it I see that the error message is "no space left on device". I have heard that people run into memory leak issues when running Docker on Mojave, but I am not sure that this problem applies to my case. Any help would be greatly appreciated!

Below is the relevant portion of the output resulting from the docker inspect command

"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "mkdir /var/lib/docker/overlay2/7af7819456a68a7a0be75ae9fafe13fd3a466bccd0f222210f6b468e0c76c0d6/merged: no space left on device",

Below is the compose file I'm using

version: "3.2"

services:

  scraper:
    image: docker-reg.myreg/simple_scraper:latest
    restart: always
    volumes:
      - /dev/shm:/dev/shm
    shm_size: '10gb'

Also, below is the relevant portion of the code where I define the webdriver

chrome_options = webdriver.ChromeOptions()

chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36')
chrome_options.add_argument('--hide-scrollbars')
chrome_options.add_argument('--enable-logging')
chrome_options.add_argument('--log-level=0')
chrome_options.add_argument('--v=99')
chrome_options.add_argument('--single-process')
chrome_options.add_argument('--ignore-certificate-errors')

self.driver = webdriver.Chrome(chrome_options=chrome_options)

Answer 1

修复它 - 原来问题是一个 chrome 选项(chrome_options.add_argument('--enable-logging')我有它正在累积内存。

Answer 2

Docker for Mac uses a Hypervisor. That hypervisor has its own disk space. You'll need to remove images from the Docker daemon to free it up

docker system prune would be a good start. Or you can reset the whole thing