在 Cloud Logging 中看不到来自 GKE 的 pod 日志

Question

I have a GKE cluster v1.22 with the Cloud Logging feature enabled. screenshot

But I can't see any application logs in the Logs Explorer.

Also, I can't see any logs for fluent-bit pod that Google uses to transfer logs to the Cloud Logging. screenshot screenshot

It has a default configmap that was created automatically with cluster creation.

API is enabled in GCP.

I can't find any related manuals and topics about it.

I will be very pleased if you help me to deal with it.

upd:

I found this in fluentbit.log on the node:

[2022/06/08 08:41:45] [error] [parser] cannot parse '0608 08:41:45.986036' after %L
[2022/06/08 08:41:45] [ warn] [parser:glog] invalid time format %m%d %H:%M:%S.%L%z for '0608 08:41:45.986036'
[2022/06/08 08:41:45] [error] [parser] cannot parse '0608 08:41:45.986083' after %L
[2022/06/08 08:41:45] [ warn] [parser:glog] invalid time format %m%d %H:%M:%S.%L%z for '0608 08:41:45.986083'
[2022/06/08 08:41:45] [error] [parser] cannot parse '0608 08:41:45.986949' after %L
[2022/06/08 08:41:45] [ warn] [parser:glog] invalid time format %m%d %H:%M:%S.%L%z for '0608 08:41:45.986949'
[2022/06/08 08:41:45] [error] [parser] cannot parse '0608 08:41:45.987048' after %L
[2022/06/08 08:41:45] [ warn] [parser:glog] invalid time format %m%d %H:%M:%S.%L%z for '0608 08:41:45.987048'

parsers conf is:

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[PARSER]
    Name        containerd
    Format      regex
    Regex       ^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[PARSER]
    Name        json
    Format      json

[PARSER]
    Name        syslog
    Format      regex
    Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
    Time_Key    time
    Time_Format %b %d %H:%M:%S

[PARSER]
    Name        glog
    Format      regex
    Regex       ^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source_file>[^ \]]+)\:(?<source_line>\d+)\]\s(?<message>.*)$
    Time_Key    time
    Time_Format %m%d %H:%M:%S.%L%z

[PARSER]
    Name        network-log
    Format      json
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Answer 1

Check if your application writes anything to stdout/stderr:
kubectl logs <pod_name>

If it returns nothing then that's why you cannot see any container logs in Logs Explorer for your application.

If it returns anything then make sure you don't have any logging exclusions potentially filtering out your application logs:
In the console menu find Operations section, then go to Logging->Logs Router
Next, on Logs Router Sink named _Default click Dots(More actions)->Edit sink
There, scroll down to 'Choose logs to filter out of sink (optional)' and make sure there are not any exclusion filters with expressions like:
resource.type=("container" OR "k8s_container")
resource.type="k8s_container" AND labels.k8s-pod/app="your_app_name"
nor any other similar matching your application.

If you find any, you can disable them (or delete them), wait a minute and check Logs Explorer again.