Docker daemon fails to process logs

Question

Docker daemon is unable to parse the json log and throws an unexpected EOF error. We are investigating the root cause of the issue.

Environment:

• Debian 4.9.189-3+deb9u2
• Docker engine 19.03.5

Docker daemon log:

Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070677515+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=345
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070695689+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=346
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070712630+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=347
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070732299+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=348
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070755016+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=349
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070773699+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=350

Docker daemon configuration

"log-driver": "json-file",
"log-opts": {
  "mode": "non-blocking",
  "max-size": "500m",
  "max-file": "3"
}

Storage space is available on partition where docker writes its logs.

Your insight in the matter is most welcome.

Answer 1

Issue=Docker Engine Error json-file: fix sporadic unexpected EOF errors

I had spent some time researching and trying to find errors in the docker logs.

There is a bug in docker engine 20.10.5 which is the version running in docker. A Github PR fixes this. https://github.com/moby/moby/pull/42104

I found the error while running

$ journalctl -u docker.service
level=warning msg="got error while decoding json" error="unexpected EOF" retries=19999

Docker driver for logs has a bug and it wont handle some log formatting. The bug then floods the system's log with retries messages. This situation generates a very annoying CPU load.

Docker engine latest is 20.10.7. The issue has been fixed in release Docker engine 20.10.6 https://docs.docker.com/engine/release-notes/#20106

Answer 2

The aforementioned issue will cause an abnormal spike in CPU usage of the docker (version <= 19) process and constantly stay at that level until you fix the logs.

To fix the logs & cpu usage, you've to truncate all the container logs (since we are unable to determine which container has faulty logs). It works only if your logs are in json format:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Please see Andre's answer for a permanent fix.