kubernetes pod 的日志究竟来自哪里（在容器级别）？

Question

I'm looking to redirect some logs from a command run with kubectl exec to that pod's logs, so that they can be read with kubectl logs <pod-name> (or really, /var/log/containers/<pod-name>.log ). I can see the logs I need as output when running the command, and they're stored inside a separate log directory inside the running container.

Redirecting the output (ie >> logfile.log ) to the file which I thought was mirroring what is in kubectl logs <pod-name> does not update that container's logs, and neither does redirecting to stdout.

When calling kubectl logs <pod-name> , my understanding is that kubelet gets them from it's internal /var/log/containers/ directory. But what determines which logs are stored there? Is it the same process as the way logs get stored inside any other docker container?

Is there a way to examine/trace the logging process, or determine where these logs are coming from?

Answer 1

Logs from the STDOUT and STDERR of containers in the pod are captured and stored inside files in /var/log/containers. This is what is presented when kubectl log is run.

In order to understand why output from commands run by kubectl exec is not shown when running kubectl log , let's have a look how it all works with an example:

First launch a pod running ubuntu that are sleeping forever:

$> kubectl run test --image=ubuntu --restart=Never -- sleep infinity

Exec into it

$> kubectl exec -it test bash

Seen from inside the container it is the STDOUT and STDERR of PID 1 that are being captured. When you do a kubectl exec into the container a new process is created living alongside PID 1:

root@test:/# ps -auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         7  0.0  0.0  18504  3400 pts/0    Ss   20:04   0:00 bash
root        19  0.0  0.0  34396  2908 pts/0    R+   20:07   0:00  \_ ps -auxf
root         1  0.0  0.0   4528   836 ?        Ss   20:03   0:00 sleep infinity

Redirecting to STDOUT is not working because /dev/stdout is a symlink to the process accessing it ( /proc/self/fd/1 rather than /proc/1/fd/1 ).

root@test:/# ls -lrt /dev/stdout
lrwxrwxrwx 1 root root 15 Nov  5 20:03 /dev/stdout -> /proc/self/fd/1

In order to see the logs from commands run with kubectl exec the logs need to be redirected to the streams that are captured by the kubelet ( STDOUT and STDERR of pid 1). This can be done by redirecting output to /proc/1/fd/1 .

root@test:/# echo "Hello" > /proc/1/fd/1

Exiting the interactive shell and checking the logs using kubectl logs should now show the output

$> kubectl logs test
Hello