列表项计数超过 1 的 GCP 日志浏览器过滤器

Question

I am trying to write a filter in GCP log explorer, which can look for a count of the values of an attribute.

Example: I am trying to find the logs like below, which has two items for "referencedTables" attribute. GCP Log Explorer Screenshot

I have tried below options which doesn't work -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.*.count>1

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.count>1

Also tried Regex looking for "tables" keyword occurrence twice -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTable=~"(\tables+::\tables+))"

Also tried Regex querying second item, which means there are more than one items -

protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables 1 =~"^[A-Za-z0-9_.]+$"

Note that - these types of logs are BigQuery audit logs, that are logged in GCP logging service, when you run "insert into.. select" type of queries in BigQuery.

Answer 1

I think you can't use logging filters to filter across log entries only within a log entry.

One solution to your problem is log-based metrics where you'd create a metric by extracting values from logs but you'd then have to use MQL to query (eg count) the metric.

A more simple (albeit ad hoc) solution is to use use gcloud logging read to --filter the logs (possibly --format the results in JSON for easier processing) and then pipeline the results into a tool like jq where you could count the results.