如何在Stackdriver Logging上禁用来自k8s_cluster的日志？

Question

We have a Google Coud Platform project where we use a cluster to deploy our apps, we have noticed on our billing this month that Stackdriver Logging increased too much so we started to create exclusions.

We excluded logs from resource.type="container" without any problem, but we are not able to do it from resource.type="k8s_cluster"

filter to create our desired exclusion:

resource.type="k8s_cluster"
protoPayload.serviceName="k8s.io"
protoPayload.resourceName="app.k8s.io/v1beta1/namespaces/default/applications/prometheus-1"

A curious thing is that Stackdriver Monitoring doesn't detect any data (0B ingested) from there, neither from our gcp-project ( https://console.cloud.google.com/logs/usage?project=xxxx ) unless we filter it using the logs viewer, then we can see the logs.

Grafana is able to detect the data with Stackdriver Datasource, you can see it here

Answer 1

The filter for resource.type="k8s_cluster" contains Audit Logs for System Events that are always enabled, hence, they won't be excluded from Stackdriver Logging unlike resource.type="container" that only has container stdout and stderr logging.

Furthermore, you can determine which logs belong to the Audit Logs stream by adding the '@type' condition to your Stackdriver filter:

resource.type="k8s_cluster"
protoPayload.serviceName="k8s.io"
protoPayload.resourceName="app.k8s.io/v1beta1/namespaces/default/applications/prometheus-1"
protoPayload."@type"="type.googleapis.com/google.cloud.audit.AuditLog"

Finally, Grafana is able to display these log-based metrics since they're exported into it via Datasource instead of going into Stackdriver.