How to see Pod logs: a container name must be specified for pod… choose one of: [wait main]

Question

I am running an Argo workflow and getting the following error in the pod's log:

error: a container name must be specified for pod <name>, choose one of: [wait main]

This error only happens some of the time and only with some of my templates, but when it does, it is a template that is run later in the workflow (ie not the first template run). I have not yet been able to identify the parameters that will run successfully, so I will be happy with tips for debugging. I have pasted the output of describe below.

Based on searches, I think the solution is simply that I need to attach "-c main" somewhere, but I do not know where and cannot find information in the Argo docs.

Describe:

Name:         message-passing-1-q8jgn-607612432
Namespace:    argo
Priority:     0
Node:         REDACTED
Start Time:   Wed, 17 Mar 2021 17:16:37 +0000
Labels:       workflows.argoproj.io/completed=false
              workflows.argoproj.io/workflow=message-passing-1-q8jgn
Annotations:  cni.projectcalico.org/podIP: 192.168.40.140/32
              cni.projectcalico.org/podIPs: 192.168.40.140/32
              workflows.argoproj.io/node-name: message-passing-1-q8jgn.e
              workflows.argoproj.io/outputs: {"exitCode":"6"}
              workflows.argoproj.io/template:
                {"name":"egress","arguments":{},"inputs":{...
Status:       Failed
IP:           192.168.40.140
IPs:
  IP:           192.168.40.140
Controlled By:  Workflow/message-passing-1-q8jgn
Containers:
  wait:
    Container ID:  docker://26d6c30440777add2af7ef3a55474d9ff36b8c562d7aecfb911ce62911e5fda3
    Image:         argoproj/argoexec:v2.12.10
    Image ID:      docker-pullable://argoproj/argoexec@sha256:6edb85a84d3e54881404d1113256a70fcc456ad49c6d168ab9dfc35e4d316a60
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Mar 2021 17:16:43 +0000
      Finished:     Wed, 17 Mar 2021 17:17:03 +0000
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:  message-passing-1-q8jgn-607612432 (v1:metadata.name)
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /mainctrfs/mnt/logs from log-p1-vol (rw)
      /mainctrfs/mnt/processed from processed-p1-vol (rw)
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-token-v2w56 (ro)
  main:
    Container ID:  docker://67e6d6d3717ab1080f14cac6655c90d990f95525edba639a2d2c7b3170a7576e
    Image:         REDACTED
    Image ID:      REDACTED
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
    Args:
    State:          Terminated
      Reason:       Error
      Exit Code:    6
      Started:      Wed, 17 Mar 2021 17:16:43 +0000
      Finished:     Wed, 17 Mar 2021 17:17:03 +0000
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/logs/ from log-p1-vol (rw)
      /mnt/processed/ from processed-p1-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-token-v2w56 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  processed-p1-vol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  message-passing-1-q8jgn-processed-p1-vol
    ReadOnly:   false
  log-p1-vol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  message-passing-1-q8jgn-log-p1-vol
    ReadOnly:   false
  argo-token-v2w56:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argo-token-v2w56
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m35s  default-scheduler  Successfully assigned argo/message-passing-1-q8jgn-607612432 to ack1
  Normal  Pulled     7m31s  kubelet            Container image "argoproj/argoexec:v2.12.10" already present on machine
  Normal  Created    7m31s  kubelet            Created container wait
  Normal  Started    7m30s  kubelet            Started container wait
  Normal  Pulled     7m30s  kubelet            Container image already present on machine
  Normal  Created    7m30s  kubelet            Created container main
  Normal  Started    7m30s  kubelet            Started container main

Answer 1

This happens when you try to see logs for a pod with multiple containers and not specify for what container you want to see the log. Typical command to see logs:

kubectl logs <podname>

But your Pod has two container, one named "wait" and one named "main". You can see the logs from the container named "main" with:

kubectl logs <podname> -c main

or you can see the logs from all containers with

kubectl logs <podname> --all-containers