如何从 StackDriver Logging API 反序列化 App Engine 应用程序日志？

Question

As part of migrating to Python 3, I need to migrate from logservice to the StackDriver Logging API . I have google-cloud-logging installed, and I can successfully fetch GAE application logs with eg:

>>> from google.cloud.logging_v2 import LoggingServiceV2Client
>>> entries = LoggingServiceV2Client().list_log_entries(('projects/projectname',),
         filter_='resource.type="gae_app" AND protoPayload.@type="type.googleapis.com/google.appengine.logging.v1.RequestLog"')
>>> print(next(iter(entries)))
proto_payload {
  type_url: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
  value: "\n\ts~brid-gy\022\0018\032R5d..."
}

This gets me a LogEntry with text application logs in the proto_payload.value field. How do I deserialize that field? I've found lots of related mentions in the docs, but nothing pointing me to a google.appengine.logging.v1.RequestLog protobuf generated class anywhere that I can use, if that's even the right idea. Has anyone done this?

Answer 1

Woo. Finally got this working. I had to generate and use the Python bindings for the google.appengine.logging.v1.RequestLog protocol buffer myself, by hand. Here's how.

First, I cloned these two repos at head:

• https://github.com/googleapis/googleapis.git
• https://github.com/protocolbuffers/protobuf.git

Then, I generated request_log_pb2.py from request_log.proto by running:

protoc -I googleapis/ -I protobuf/src/ --python_out . googleapis/google/appengine/logging/v1/request_log.proto

Finally, I pip installed googleapis-common-protos and protobuf . I was then able to deserialize proto_payload with:

from google.cloud.logging_v2 import LoggingServiceV2Client
client = LoggingServiceV2Client(...)
log = next(iter(client.list_log_entries(('projects/brid-gy',),
   filter_='logName="projects/brid-gy/logs/appengine.googleapis.com%2Frequest_log"')))

import request_log_pb2
pb = request_log_pb2.RequestLog.FromString(log.proto_payload.value)
print(pb)

Answer 2

You can use the LogEntry.to_api_repr() function to get a JSON version of the LogEntry .

>>> from google.cloud.logging import Client
>>> entries = Client().list_entries(filter_="severity:DEBUG")        
>>> entry = next(iter(entries))
>>> entry.to_api_repr()
{'logName': 'projects/PROJECT_NAME/logs/cloudfunctions.googleapis.com%2Fcloud-functions'
, 'resource': {'type': 'cloud_function', 'labels': {'region': 'us-central1', 'function_name': 'tes
t', 'project_id': 'PROJECT_NAME'}}, 'labels': {'execution_id': '1zqolde6afmx'}, 'insertI
d': '000000-f629ab40-aeca-4802-a678-d513e605608e', 'severity': 'DEBUG', 'timestamp': '2019-10-24T2
1:55:14.135056Z', 'trace': 'projects/PROJECT_NAME/traces/9c5201c3061d91c2b624abb950838b4
0', 'textPayload': 'Function execution started'}

Answer 3

Do you really want to use the API v2?

If not, use from google.cloud import logging and set os.environ['GOOGLE_CLOUD_DISABLE_GRPC'] = 'true' - or similar env setting.

That will effectively return a JSON in payload instead of payload_pb

Answer 4

You can try to use: - gcloud beta logging read ${LOG_FILTER}

Or

You could change the approach for writing python scripts. You could use the python subprocess module to call the gcloud command (keeping in mind that this is part of the beta command group, which can change on updates without warning. Sending the output into a string, and then parsing the entries that way. This, while it would mean you'd have your logs in a python environment (with all the leverage that brings) however, it wouldn't use the google.cloud library.