A lot of kubelet errors : Failed to update stats for container

Question

I've the a lot of error logs reported by kubelet :

Sep 07 09:43:51 kubenode-1 kubelet[10320]: I0907 09:43:51.651224   10320 container.go:369] Failed to update stats for container "/docker/01ad0eff434033752c1f39944e9965e38a07081fcbfe26dc35358bb63be18082": failed to read stat from "/sys/class/net/veth2fc2d33/statistics/rx_bytes" for device "veth2fc2d33", continuing to 
Sep 07 09:43:56 kubenode-1 kubelet[10320]: I0907 09:43:56.051022   10320 container.go:369] Failed to update stats for container "/": failed to read stat from "/sys/class/net/calic1976c4e52f/statistics/rx_bytes" for device "calic1976c4e52f", continuing to push stats

I don't know what is the problem exactly. Please find below further information

docker info
Containers: 27
Images: 121
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 175
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 2
Total Memory: 2.95 GiB
Name: kubenode-1
ID: LXO4:TD3E:ZAL5:AUWE:PN6W:KFZX:S4QR:AX6V:776M:VHVT:7Z3O:O72V
Username: smaine
Registry: [https://index.docker.io/v1/]

kubelet --version
Kubernetes v1.0.3

I'm running the kubernetes cluster on debian jessie.

Furthermore i have these errors :

Sep 07 09:37:19 kubenode-1 kubelet[10320]: W0907 09:37:19.148453   10320 manager.go:1161] No ref for pod '814c1a33de45655e8cff2044485913ab568b9ab858ed2c5aa30d0034b82a6660'
Sep 07 09:37:29 kubenode-1 kubelet[10320]: W0907 09:37:29.265237   10320 manager.go:1161] No ref for pod '06a3e276f8b3dca0c3ea20b5feee4ab9b5ee97ef44aad1aef2f0102d5ddfa40c'
Sep 07 09:37:31 kubenode-1 kubelet[10320]: W0907 09:37:31.065126   10320 manager.go:1161] No ref for pod '686039f754799616a0957d3fcc516bc46b9601a2c6ac304a010397142fc06cd0'
Sep 07 09:37:31 kubenode-1 kubelet[10320]: W0907 09:37:31.075288   10320 kubelet.go:1343] Orphaned volume "c2f1c714-52f8-11e5-b108-5254000ec1d7/influxdb-persistent-storage" found, tearing down volume
Sep 07 09:37:31 kubenode-1 kubelet[10320]: W0907 09:37:31.914188   10320 kubelet.go:1343] Orphaned volume "c2f1c714-52f8-11e5-b108-5254000ec1d7/default-token-v2j1u" found, tearing down volume
Sep 07 09:38:49 kubenode-1 kubelet[10320]: E0907 09:38:49.224758   10320 kubelet.go:682] Image garbage collection failed: failed to find information for the filesystem labeled "docker-images"

I'm using calico as network overlay but i don't think it's related. Please let me know if you need more info.

Regards,

Smana

Answer 1

If you use Docker v1.7 or above, you may have run into a cadvisor bug, where network stats are not collected correctly. See https://github.com/kubernetes/kubernetes/issues/13189

The bug has already being fixed in cadvisor, and kubernetes will include the fix soon.