用于 CLI 事件、AWS Kinesis + Lambda 的 Python 中的 UnicodeDecodeError

Question

I have AWS Kinesis stream (with server-side encryption) set up, with Lambda function for event handling. I am trying to test this from CLI and get the error.

CLI for inserting event:

aws kinesis put-record \
--stream-name "dev" \
--partition-key "user_event" \
--data '
    [{
        "user_id": 2002,
        "group_id": 2002,
        "created_at": "2021-11-24T16:03:03Z",
        "user_event_id": 989898989898,
        "event_type": "in",
        "event_date": "2022-11-03T14:03:03Z",
        "category": "music",
        "is_free": true
    }]' \
--profile dev \
--region eu-central-1

Parsing code which throws the error:

for record in kinesis_data["Records"]:
    partition_key = record["kinesis"]["partitionKey"]

    if not self.accepted_parsers(partition_key):
        continue

    encoded_data = record["kinesis"]["data"]
    payload_string = base64.b64decode(encoded_data)

    payload = json.loads(payload_string)

Error itself:

[ERROR] UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 0: invalid start byte
Traceback (most recent call last):
  File "/var/task/events_processing.py", line 38, in lambda_handler
    status_code = lambda_interactor.call(event)
  File "/var/task/lib/lambda_interactor.py", line 113, in call
    event_dicts = self.kinesis_parser.call(payload)
  File "/var/task/lib/kinesis_parser.py", line 22, in call
    payload = json.loads(payload_string)
  File "/var/lang/lib/python3.8/json/__init__.py", line 343, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')

I tried printing to debug:

• encoded_data : userid2002groupid2002createdat20211124T160303Zusereventid989898989898eventtypeineventdate20221103T140303Zcategorymusicisfreetrue

• payload_string : b"\x99\xe9\x9bz\xb8\x9d\xdbM6r\x89\xa6\xbax\xad\xca'v\xd3M\x9c\xad\xe6\xady\xd6\xad\xdbM\xb5\xd7]\xb8O^\xb4\xdfM\xd9z\xf7\xa7\xb5\xba\xe2\xb5\xe7\xafz{bw\xdf=\xf3\xdf=\xf3\xdf=\xf1\xeb\xde\x9e\xdbr\xa5\xe8\xa7z\xf7\xa7\xb5\xd6\xad{m6\xdb]t\xdd=x\xd3}7e\xc6\xadz\n+\xcak\xac\x89\xc8\xac~\xb7\x9e\xb6\xbb\x9e"

How can I fix this? I followed examples from AWS tutorials, and this exact setup used to work properly for months, and stopped working about a month ago with no changes in code. I see that special characters were all removed, but I have no idea why.

Answer 1

I found out - AWS CLI v2 has undocumented behavior changes. I had to add the following options to the CLI command:

--region eu-central-1 \
--cli-binary-format raw-in-base64-out

The latter is especially important, as only this way the proper base64-encoded message is sent and received.