AWS EC2 Launch logs in cloudwatch windows 2016 image

Question

I'm trying to forward the EC2 Launch logs to cloudwatch from my win 2016-based EC2 instance.

For some reason I can't see the log groups for this specific category.

Here's example of my AWS.EC2.Windows.CloudWatch.json :

{
  "IsEnabled": true,
  "EngineConfiguration": {
    "PollInterval": "00:00:15",
    "Components": [
      {
        "Id": "Ec2Config",
        "FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Parameters": {
          "LogDirectoryPath": "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Log",
          "TimestampFormat": "yyyy-MM-ddTHH:mm:ss.fffZ:",
          "Encoding": "UTF-8",
          "Filter": "UserdataExecution.log",
          "CultureName": "en-US",
          "TimeZoneKind": "UTC"
        }
      },
      {
        "Id": "EC2ConfigSink",
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Parameters": {
          "Region": "eu-west-1",
          "LogGroup": "/my-customer/deployment/ec2config-userdata",
          "LogStream": "ec2config-userdata"
        }
      }
      ...
      I have a few more definitions in this file
      ...],
    "Flows": {
      "Flows":
      [
        "Ec2Config,EC2ConfigSink",
        ... other references here
      ]
    }
 }

Cloudwatch agent starts and doesn't report any errors, I can see data from other sources (some application log files - I skipped the definitions intentionally)

It means the cloudwatch config file is correct and is applied / placed in a correct directory.

Logs are coming through with no problem except for the EC2 launch logs.

I'm wondering if anybody ran into this problem? It works perfectly on Windows 2012 - based images

Answer 1

Apparently, the SSM Agent starts after the EC2 Launch executes UserData script. I can see it from the SSM Agent's log file modification timestamps.

Therefore, there's no log forwarding happening during the EC2 Launch . When the SSM Agent starts and loads the cloudwatch plugin, the log files are already filled with entries and never change (wallpaper log is the only exception) So they never end up in cloudwatch console.

There's been a lot of changes implemented on AWS side: they switch to .Net core, removed EC2 config service and moved the log forwarding logic to SSM Agent (cloudwatch Plugin) for Windows 2016-based AMIs

It looks like the behavior has changed quite significantly too so there's no way to get the EC2 launch logs in cloudwatch (when using AWS toolset-only)

Basically we have to stick to our Application logs only which is very unfortunate. We rely on EC2 launch logs to see if the instance started & successfully executed user data.