When logging to CloudWatch logs in a JSON format what is the name of the timestamp property and what is the expected format?

Question

I'm logging to CloudWatch logs using a logging framework that sends JSON. At the moment every log in CloudWatch is being logged with the same date.

What is the name of the JSON property that CloudWatch Logs looks for to determine the date/time of the logged event and what format does it have to be in to be logged correctly.

Are these details documented by AWS and if so, where? Cause I can't find it within the CloudWatch logs documentation...

Here's my example log at the moment:

{
    "time": "2022-02-16 19:11:29.9734",
    "level": "ERROR",
    "message": "Something went wrong...",
    "callsite": "My.NameSpace.Services.ImageDetector",
    "logger": "My.NameSpace.Services.ImageDetector",
    "url": "https://example.com",
    "action": "CheckImage",
    "ipaddress": "8.8.8.8.8",
    "user": "root"
}

But it's logging with the datetime 2022-02-14T01:12:47.160+00:00 every time.

Or can it only be sent along with the API call and is not pulled out of the log contents itself as I think it is?

Answer 1

I assume you use the latest CloudWatch Agent . Please, check the CloudWatch agent configuration file: Logs section .

The agent reads entries from the log file one by one. It then uses the timestampFromLogLine() function to scan the log entry string with RegExp and extract and parse the timestamp.

I think the RegExp for you might be the following.

"timestamp_format": "\"time\": \"%Y-%m-%d %H:%M:%S.%f\"",
"timezone": "local"

Remember to specify the timezone because you do not have it in the timestamp.

Side note: You show log entry as multiline. You might have reformated it for readability when you published it. If it appears like this in the log, you might need the multi_line_start_pattern configuration parameter.

For reference, here is my complete configuration file to parse Ghost logs.

{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/www/victorsmirnov.blog/content/logs/https___victorsmirnov_blog__production.log",
            "log_group_name": "victorsmirnov.blog/ghost-access-logs/{instance_id}",
            "timezone": "UTC",
            "timestamp_format": "\"time\":\"%Y-%m-%dT%H:%M:%S.%fZ\""
          },
          {
            "file_path": "/var/www/victorsmirnov.blog/content/logs/https___victorsmirnov_blog__production.error.log",
            "log_group_name": "victorsmirnov.blog/ghost-error-logs/{instance_id}",
            "timezone": "UTC",
            "timestamp_format": "\"time\": \"%Y-%m-%d %H:%M:%S.%f\""
          }
        ]
      }
    }
  }
}

Example of the log line

{"name":"Log","hostname":"ip-10-100-12-1","pid":21129,"level":30,"req":{"meta":{"requestId":"157ece9f-ed91-412b-9cf1-695a97d0aebd","userId":null},"url":"/health","method":"GET","originalUrl":"/health","params":{},"headers":{"host":"10.100.12.1:2369","connection":"close","user-agent":"ELB-HealthChecker/2.0","accept-encoding":"gzip, compressed"},"query":{}},"res":{"_headers":{"x-powered-by":"Express","cache-control":"public, max-age=31536000","location":"https://10.100.12.1:2369/health/","vary":"Accept, Accept-Encoding","content-type":"text/plain; charset=utf-8","content-length":"66"},"statusCode":301,"responseTime":"1ms"},"msg":"","time":"2022-08-04T00:00:27.348Z","v":0}

The @timestamp column generated by CloudWatch matches the time property parsed from the JSON record.

Answer 2

It seems that there is a concept of "discovered fields" when posting JSON to CloudWatch. One of these is the log timestamp.

If you post a timestamp field in your JSON payload it will auto detect the timestamp from the value.

This can be confirmed by querying for log entries in CloudWatch insights. The log entry only included a "timestamp" field, but the auto generated @timestamp value is filled:

Please note that the timestamp display in CloudWatch displays the timestamp in your local timezone:

Discovered fields are documented here .