如何在 python 中使用 boto3 过滤 cloudwatch 日志

Question

I'd like to filter the logs from API Gateway with Cloudwatch.

Here is a sample of my logs:

(f810f3b1-5aqa-4af1-be31-bq10d3w99fqp) Endpoint request body after transformations: {"domain":"example.com"}
(f810f3b1-5aqa-4af1-be31-bq10d3w99fqp) HTTP Method: POST, Resource Path: /v/
(f810f3b1-5aqa-4af1-be31-bq10d3w99fqp) API Key: **************
(f810f3b1-5aqa-4af1-be31-bq10d3w99fqp) Method request path: {}
(f810f3b1-5aqa-4af1-be31-bq10d3w99fqp) Method request query string: {0.49120039624=}

My goal is to filter only the @message containing the request body to get the domain from it using boto3. Here's my code:

query = "fields @timestamp, @message | filter @message in ['domain']"

response_query = client.start_query(
        logGroupName=log_group, 
        startTime=int((datetime.now() - timedelta(hours=5)).timestamp()),
        endTime=int(datetime.now().timestamp()),
        queryString=query,
        limit=1000
    )

=> {'results': [], 'statistics': {....} I'm getting no result from this query, do you have any idea why?

Answer 1

The solution is to use like operator for fuzzy match. in operator in CloudWatch query is similar to it in other languages like Python,

>>> 'a' in ['a', 'b']
True

in only checks for exact matches. Its typical usage in CloudWatch is to check low-cardinality set membership in the discovered log fields. For example, the discovered log field @type in Lambda logs indicates the type of a log message in a lambda invocation. The possible values are START , END , REPORT . In other words, the cardinality of this set is 3, which is pretty low. Then I can use the following query to get a glance of how the duration and max memory usage look like in the recent invocations.

fields @timestamp, @message
| filter @type in ['REPORT']
| sort @timestamp desc