TypeError 从云端将结构化日志写入谷歌云日志 function

Question

I am trying to write a log from a cloud function into cloud logging, but I am having trouble doing this. My code is the following, where the message is a log like this: "20210319-11:39:58-CET - INFO - this is an info message" and the atttributes are simply strings.

import base64
import json
import os
import pytz
from datetime import datetime
from google.cloud import logging
from google.cloud.logging_v2.resource import Resource

RESOURCE_TYPE = "cloudiot_devices"
PROJECT = os.environ.get("GCP_PROJECT")
DEVICE_REGISTRY = REGISTRY_ID
LOCATION = LOCATION
DATEFORMAT = "%Y%m%d-%H:%M:%S"

def iot_logger(event, context):
    """Triggered from a message on a Cloud Pub/Sub topic.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    log = base64.b64decode(event['data']).decode('utf-8')
    attributes = event['attributes']

    print(log)
    print(json.dumps(attributes))

    print("preparing timestamp")
    # Extract elements from log
    date_str,severity,message = log.split(" - ")

    # This is the main content for the log
    log_content = {"message":message}

    # turn log timestamp str to an aware datetime object for our timestamp
    tzname = date_str.split("-")[-1]
    # remove time zone to avoid tripping up the datetime module
    date_str = date_str[:-4]
    date_unaware = datetime.strptime(date_str,DATEFORMAT)
    date_aware = date_unaware.astimezone(pytz.timezone(tzname))

    print(f"done preparing timestamp: {date_aware}")

    # extract attributes
    logger_name = attributes["logger_name"]
    device_num_id = attributes["device_num_id"]

    print("creating resource")
    # create resource 
    resource_labels = {
        'project_id':PROJECT,
        'device_num_id':device_num_id,
        'device_registry_id':DEVICE_REGISTRY,
        'location':LOCATION
    }
    log_resource = Resource(type=RESOURCE_TYPE,labels=resource_labels)

    print("initiating client")
    # initiate logging client
    log_client = logging.Client()

    logger = log_client.logger(logger_name)

    print("creating structured log")
    logger.log_struct(log_content,severity=severity,resource=log_resource,timestamp=date_aware)

However, I keep facing issues with the log_struct command and I haven't been able to figure out what is the problem. Initially I was passing a string to the timestamp and it wouldn't work, so now I am passing a date object. But now the error is different: EDIT: There's more than one:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 589, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 603, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object. 

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 589, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 603, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object. 

"Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 149, in view_func
    function(data, context)
  File "/workspace/main.py", line 64, in iot_logger
    logger.log_struct(log_content,severity=severity,resource=log_resource,timestamp=date_aware)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 180, in log_struct
    self._do_log(client, StructEntry, info, **kw)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 133, in _do_log
    client.logging_api.write_entries([api_repr])
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 140, in write_entries
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 140, in <listcomp>
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 507, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 454, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 597, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse resource field: Failed to parse labels field: expected string or bytes-like object.."

It says it needs a string or bytes-like object, but I am not sure where. I imagine it's not related to the date, as it was giving me an error when it was a string. Also the resource I believe should work, I am looking at a very similar example on this stackoverflow question, but it doesn't work for me:

Google Cloud Functions Python Logging issue

What am I doing wrong?

Answer 1

I solved it with the help of @jabbson. I ran into a few issues, but mostly I was giving it a couple of wrong things without realizing it.

I was using the environment variables as pointed out here: https://cloud.google.com/functions/docs/env-var#nodejs_8_python_37_and_go_111 but for some reason I was getting null for project id. Also I had a typo in my resource_type.

Lastly, I had to play around with the timestamp until it actually worked like I wanted it to. The LogEntry documentation says it's a string, but in this case I had to pass it a datetime object and converting local time to UTC so I could actually find it in cloud-logging.