如何从 Kubernetes pod 到本地运行的 Wireshark 的 pipe 实时 pcap 记录？

Question

I'm attempting to get a view in Wireshark of live network traffic in one of my Kubernetes pods. In plain old Docker, I was able to run this:

docker run --rm --net=container:app_service_1 crccheck/tcpdump -i any --immediate-mode -w - | wireshark -k -i -

This spins up a simple container that runs tcpdump with the arguments shown, and pipes the packet captures in pcap format to stdout (the -w - argument). This output is then piped to Wireshark running on my host machine, which displays the packets as they arrive.

How do I do something similar in Kubernetes?

I've tried applying a patch as follows:

  template:
    spec:
      containers:
        - name: tcpdumper
          image: crccheck/tcpdump
          args: ["-i", "any", "--immediate-mode", "-w", "-"]
          tty: true
          stdin: true

And I apply this by running k attach -it app-service-7bdb7798c5-2lr6q | wireshark -k -i - k attach -it app-service-7bdb7798c5-2lr6q | wireshark -k -i -

But this doesn't seem to work; Wireshark starts up but it immediately shows an error:

Data written to the pipe is neither in a supported pcap format nor in pcapng format

Answer 1

I haven't used k8s a lot, but the docker run gets the entire clean stdout, while I get the impression that k attach doesn't.

I don't think that kubectl has an equivalent of docker run, that gives you clean stdout, but you might be able to do something with kubectl exec.

A possible test would be to redirect the output to a file, and see if it's valid output for the command you're running, and that there's nothing unexpected there.

Answer 2

I highly suggest you to read Using sidecars to analyze and debug network traffic in OpenShift and Kubernetes pods .

This article explains why you cant read traffic data directly from a pod and gives you an alternative on how to do it using a sidecar.

In short words, the containers most likely run on an internal container platform network that is not directly accessible by your machine.

A sidecar container is a container that is running in the same pod as the actual service/application and is able to provide additional functionality to the service/application.

TCPdump effectively in Kubernetes is a bit tricky and requires you to create a side car to your pod. What you are facing is actually the expected behavior.

run good old stuff like TCPdump or ngrep would not yield much interesting information, because you link directly to the bridge network or overlay in a default scenario.

The good news is, that you can link your TCPdump container to the host network or even better, to the container network stack. Source: How to TCPdump effectively in Docker

The thing is that you have two entry points, one is for nodeIP:NodePort the second is ClusterIP:Port. Both are pointing to the same set of randomization rules for endpoints set on kubernetes iptables.

As soon as it can happen on any node it's hard to configure tcpdump to catch all interesting traffic in just one point.

The best tool I know for such kind of analysis is Istio, but it works mostly for HTTP traffic.

Considering this, the best solution is to use a tcpdumper sidecar for each pod behind the service.

Let's go trough an example on how to achieve this

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web-app
        image: nginx
        imagePullPolicy: Always        
        ports:
        - containerPort: 80
          protocol: TCP
      - name: tcpdumper
        image: docker.io/dockersec/tcpdump
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: web-svc
  namespace: default
spec:
  ports:
  - nodePort: 30002
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  type: NodePort

On this manifest we can notice tree important things. We have a nginx container and one tcpdumper container as a side car and we have a service defined as NodePort.

To access our sidecar, you have to run the following command:

$ kubectl attach -it web-app-db7f7c59-d4xm6 -c tcpdumper

Example:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        13d
web-svc      NodePort    10.108.142.180   <none>        80:30002/TCP   9d

$ curl localhost:30002
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

$ kubectl attach -it web-app-db7f7c59-d4xm6 -c tcpdumper
Unable to use a TTY - container tcpdumper did not allocate one
If you don't see a command prompt, try pressing enter.
> web-app-db7f7c59-d4xm6.80: Flags [P.], seq 1:78, ack 1, win 222, options [nop,nop,TS val 300957902 ecr 300958061], length 77: HTTP: GET / HTTP/1.1
12:03:16.884512 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.1336: Flags [.], ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 0
12:03:16.884651 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.1336: Flags [P.], seq 1:240, ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 239: HTTP: HTTP/1.1 200 OK
12:03:16.884705 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.1336: Flags [P.], seq 240:852, ack 78, win 217, options [nop,nop,TS val 300958061 ecr 300957902], length 612: HTTP
12:03:16.884743 IP 192.168.250.64.1336 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 240, win 231, options [nop,nop,TS val 300957902 ecr 300958061], length 0
12:03:16.884785 IP 192.168.250.64.1336 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 852, win 240, options [nop,nop,TS val 300957902 ecr 300958061], length 0
12:03:16.889312 IP 192.168.250.64.1336 > web-app-db7f7c59-d4xm6.80: Flags [F.], seq 78, ack 852, win 240, options [nop,nop,TS val 300957903 ecr 300958061], length 0
12:03:16.889351 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.1336: Flags [F.], seq 852, ack 79, win 217, options [nop,nop,TS val 300958062 ecr 300957903], length 0
12:03:16.889535 IP 192.168.250.64.1336 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 853, win 240, options [nop,nop,TS val 300957903 ecr 300958062], length 0
12:08:10.336319 IP6 fe80::ecee:eeff:feee:eeee > ff02::2: ICMP6, router solicitation, length 16
12:15:47.717966 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [S], seq 3314747302, win 28400, options [mss 1420,sackOK,TS val 301145611 ecr 0,nop,wscale 7], length 0
12:15:47.717993 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.2856: Flags [S.], seq 2539474977, ack 3314747303, win 27760, options [mss 1400,sackOK,TS val 301145769 ecr 301145611,nop,wscale 7], length 0
12:15:47.718162 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 1, win 222, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.718164 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [P.], seq 1:78, ack 1, win 222, options [nop,nop,TS val 301145611 ecr 301145769], length 77: HTTP: GET / HTTP/1.1
12:15:47.718191 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.2856: Flags [.], ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 0
12:15:47.718339 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.2856: Flags [P.], seq 1:240, ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 239: HTTP: HTTP/1.1 200 OK
12:15:47.718403 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.2856: Flags [P.], seq 240:852, ack 78, win 217, options [nop,nop,TS val 301145769 ecr 301145611], length 612: HTTP
12:15:47.718451 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 240, win 231, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.718489 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 852, win 240, options [nop,nop,TS val 301145611 ecr 301145769], length 0
12:15:47.723049 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [F.], seq 78, ack 852, win 240, options [nop,nop,TS val 301145612 ecr 301145769], length 0
12:15:47.723093 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.2856: Flags [F.], seq 852, ack 79, win 217, options [nop,nop,TS val 301145770 ecr 301145612], length 0
12:15:47.723243 IP 192.168.250.64.2856 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 853, win 240, options [nop,nop,TS val 301145612 ecr 301145770], length 0
12:15:50.493995 IP 192.168.250.64.31340 > web-app-db7f7c59-d4xm6.80: Flags [S], seq 124258064, win 28400, options [mss 1420,sackOK,TS val 301146305 ecr 0,nop,wscale 7], length 0
12:15:50.494022 IP web-app-db7f7c59-d4xm6.80 > 192.168.250.64.31340: Flags [S.], seq 3544403648, ack 124258065, win 27760, options [mss 1400,sackOK,TS val 301146463 ecr 301146305,nop,wscale 7], length 0
12:15:50.494189 IP 192.168.250.64.31340 > web-app-db7f7c59-d4xm6.80: Flags [.], ack 1, win 222, options 

You can also take a look at ksniff tool, a kubectl plugin that utilize tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster.