从 Cloud Functions 中的 PubSub 日志消息中提取 textPayload

Question

I have created a workflow that sends some logs to Pub/Sub. These logs trigger a Cloud Function which process the logs and extract the textPayload, but that part it's failing.

Cloud Function code

def main(event, context):
    print("cf-dataflow-status-checker - main - start")
    """Triggered from a message on a Cloud Pub/Sub topic.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    pubsub_message = base64.b64decode(event['data']).decode('utf-8')

    print(pubsub_message)
    text_payload = pubsub_message['jsonPayload']['textPayload']
    print(text_payload)
    job_name = pubsub_message['jsonPayload']['resource']['labels']['job_name']
    print(job_name)

    if SUMMARY_ERROR_LOG in text_payload:
        message = 'Dataflow pipeline has failed.'
        proc_id = find_process(job_name)
        print('proc_id: {}'.format(proc_id))
        end_process_error(proc_id, message)
        return {'status': 'KO'}

    print("cf-dataflow-status-checker - main - end")
    return {'status': 'OK'}

The line

text_payload = pubsub_message['jsonPayload']['textPayload']

and

job_name = pubsub_message['jsonPayload']['resource']['labels']['job_name']

are causing the following errors:

text_payload = json_message['jsonPayload']['textPayload']
KeyError: 'jsonPayload'

Pub / Sub Message

pubsub_message = {
    "insertId": "000002-e8d21b19-1be9-4f76-8359-2c4bc8d5b451",
    "jsonPayload": {
        "insertId": "11qnus8bu4",
        "timestamp": "2022-01-11T18:15:52.85415621Z",
        "receiveTimestamp": "2022-01-11T18:15:53.543572717Z",
        "textPayload": "Workflow failed. Causes: S11:Merge/_CoGBKImpl/GroupByKey/Read+Merge/_CoGBKImpl/GroupByKey/GroupByWindow+Merge/_CoGBKImpl/MapTuple(collect_values)+Merge/RestoreTags+Process the data+Parse result 1+Parse result 2+Write to txt 1/Write/WriteImpl/Map(<lambda at iobase.py:1126>)+Write to txt 1/Write/WriteImpl/WindowInto(WindowIntoFn)+Write to txt 1/Write/WriteImpl/GroupByKey/Reify+Write to txt 1/Write/WriteImpl/GroupByKey/Write+Write to txt 2/Write/WriteImpl/Map(<lambda at iobase.py:1126>)+Write to txt 2/Write/WriteImpl/WindowInto(WindowIntoFn)+Write to txt 2/Write/WriteImpl/GroupByKey/Reify+Write to txt 2/Write/WriteImpl/GroupByKey/Write failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. Root cause: Work item failed.",
        "labels": {
            "dataflow.googleapis.com/region": "us-central1",
            "dataflow.googleapis.com/job_name": "jobname-test-01/11/2022-18:11:31",
            "dataflow.googleapis.com/job_id": "2022-01-11_10_11_31-8608721464069559920",
            "dataflow.googleapis.com/log_type": "system"
        },
        "resource": {
            "labels": {
                "region": "us-central1",
                "job_name": "jobname-test-01/11/2022-18:11:31",
                "step_id": "",
                "project_id": "{PROJECT_ID}",
                "job_id": "2022-01-11_10_11_31-8608721464069559920"
            },
            "type": "dataflow_step"
        },
        "logName": "projects/{PROJECT_ID}/logs/dataflow.googleapis.com%2Fjob-message"
    },
    "resource": {
        "type": "cloud_function",
        "labels": {
            "region": "europe-west1",
            "project_id": "{PROJECT_ID}",
            "function_name": "cf-dataflow-status-checker"
        }
    },
    "timestamp": "2022-01-11T18:16:01.692Z",
    "severity": "ERROR",
    "labels": {
        "execution_id": "aoov0zbhutdn"
    },
    "logName": "projects/{PROJECT_ID}/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
    "trace": "projects/{PROJECT_ID}/traces/d149bc4fa2472054cf3fe0ec19e2b545",
    "receiveTimestamp": "2022-01-11T18:16:02.110779678Z"
}

I have copied the json to my code editor and it works perfectly...

Anyone knows whats happening?

Answer 1

Your pubsub_message is a string and needs to be parsed as JSON. Use something like the following:

pubsub_message = json.loads(base64.b64decode(event['data']))