How to view container logs via stackdriver on GKE

Question

I've read that by default, clusters on GKE write all logs to stackdriver. On the GKE UI, I can see the container logs of all my deployments, but am looking to get these logs from an API programmatically.

I've tried to do something like gcloud logging read "resource.labels.pod_id="<pod-id>"" but nothing is returned.

Do I need to do something special in order to enable logging on the cluster? And if not, how do I access logs for a specific deployment/pod/container?

Further, are these logs persistent? As in, when the deployment dies, can I still access these logs?

Answer 1

You have several options to read the GKE pods logs from Stackdriver. Some of them are:

Reading logs

To read log entries in Logging, you can do any of the following:

• Use the Logs Viewer in the Google Cloud Console.
• Call the Logging API through the Client Libraries for your programming language.
• Call the Logging API REST endpoints directly. See the Logging API reference documentation.

• Use the Cloud SDK. For more information, see the gcloud logging command-line interface.

  Cloud.google.com: Logging: Setup

As for:

Do I need to do something special in order to enable logging on the cluster? And if not, how do I access logs for a specific deployment/pod/container?

Please refer to: Cloud.google.com: Logging: Access control

Answering below:

Further, are these logs persistent? As in, when the deployment dies, can I still access these logs?

Yes you can still access this logs, even if the deployment was deleted. You can still access the logs even if you delete your cluster. Logs stored in Stackdriver have retention policies which will store the logs for set amount of time. Please refer to:

• Cloud.google.com: Logging: Quotas
• Cloud.google.com: Logging: Storage

---

Please take a look on the example below which shows how to access logs with gcloud command:

Steps:

• Create a Deployment which will send logs to Stackdriver
• Check if the logs are stored in Stackdriver
• Get the logs from Stackdriver with gcloud

Create a Deployment

Please follow a Google Cloud Platform guide to spawn a Deployment which will send data to Stackdriver:

• Cloud.google.com: Kubernetes Engine: Custom Metrics autoscaling: Before you begin - start
• Cloud.google.com: Kubernetes Engine: Custom Metrics autoscaling: Exporting - end

Check if the logs are in Stackdriver.

Logs exported by above deployment will be stored in Stackdriver. The example of it should look like this:

{
 insertId: "REDACTED"  
 labels: {
  k8s-pod/pod-template-hash: "545464fb5"   
  k8s-pod/run: "custom-metric-sd"   
 }
 logName: "projects/REDACTED/logs/stderr"  
 receiveTimestamp: "2020-05-26T10:17:16.161949129Z"  
 resource: {
  labels: {
   cluster_name: "gke-logs"    
   container_name: "sd-dummy-exporter"    
   location: "ZONE"    
   namespace_name: "default"    
   pod_name: "custom-metric-sd-545464fb5-2rdvx"    
   project_id: "REDACTED"    
  }
  type: "k8s_container"   
 }
 severity: "ERROR"  
 textPayload: "2020/05/26 10:17:10 Finished writing time series with value: 0xc420015290
"  
 timestamp: "2020-05-26T10:17:10.356684667Z"  
}

Above log entry will help with creating a gcloud command to get only specified logs from the Stackdriver.

Get the logs from Stackdriver with gcloud

As you pointed:

I've tried to do something like gcloud logging read "resource.labels.pod_id=""" but nothing is returned.

The fact that nothing is returned is most probably connected with the fact that the requested resource was not found with the command you used.

To get this logs from Stackdriver invoke below command:

$ gcloud logging read "resource.type=k8s_container AND resource.labels.container_name=sd-dummy-exporter" 

Dividing above command on smaller pieces:

• resource.type=k8s_container - it will get the logs with a type of k8s_container
• resource.labels.container_name=XYZ - it will get the logs with a specified container_name .

This pieces are directly connected with example singular log entry mentioned earlier.

A tip:

resources.labels.container_name can be used to collect logs from multiple pods as the specific pod can be referenced with pod_name .