Disk space issue on docker container

Question

We have deployed jenkins on docker container and recently we started seeing that our jenkins server is not coming up due to disk space issue. Below is the error we see in logs.

2022-09-17 21:41:32.567+0000 [id=32]    INFO    hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect V3LOCITY-SLAVE-02
/usr/local/bin/jenkins.sh: line 38: cannot create temp file for here-document: No space left on device
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Exception in thread "main" java.io.IOException: Jenkins has failed to create a temporary file in /tmp
        at Main.extractFromJar(Main.java:498)
        at Main._main(Main.java:310)
        at Main.main(Main.java:151)
Caused by: java.io.IOException: No space left on device
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2063)
        at Main.extractFromJar(Main.java:495)
        ... 2 more

We assume issue with docker container running of out space, See below info for your reference.

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1                   1                   572.5MB             0B (0%)
Containers          1                   0                   9.467GB             9.467GB (100%)
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

Assuming container running of space we have increased it to 40 GB by adding below content in /etc/docker/daemon.json file and recreated the contained but still see the same issue after restart of container

{
        "storage-driver": "devicemapper",
        "storage-opts": [
                "dm.basesize=40G"
        ]
}

See below docker info your reference.

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 19.03.11-ol
 Storage Driver: devicemapper
  Pool Name: docker-249:0-1140851221-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 42.95GB
  Backing Filesystem: xfs
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Data Space Used: 10.82GB
  Data Space Total: 107.4GB
  Data Space Available: 96.56GB
  Metadata Space Used: 6.877MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.141GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.170-RHEL7 (2020-03-24)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: 52de29d
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.1.12-124.65.1.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.56GiB
 Name: vm-app-docker-jenkinsqa
 ID: TAII:OWLM:Y3BU:65DC:A3SK:SSJQ:H6H2:BLA2:HQA5:ODCP:Y7S5:KCJ2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Registries:

Answer 1

You need to map jenkins home to an external folder (volume) and make sure the host has enough space.

See Jenkins docs for more details.

For example:

docker run --name jenkins -v /var/jenkins_home:/var/jenkins_home ...