Amazon CloudWatch Agent not sending OnPremise metrics/logs

Question

I'm trying to connect my on-premise instance to AWS via CloudWatch Agent to record system metrics. Installed latest version from .deb package and nightly version from .deb package and results are the same (except that nightly version runs via systemd).

Even though logging is enabled both for agent and SDK nothing appears in the logs.

After launching via ctl script it is launched but in fact service daemon is constantly restarted; when started with service amazon-cloudwatch-agent start it is reported as failed:

Apr 22 00:20:56 sero1 systemd[1]: Started Amazon CloudWatch Agent.
Apr 22 00:21:02 sero1 start-amazon-cloudwatch-agent[7614]: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
Apr 22 00:21:02 sero1 start-amazon-cloudwatch-agent[7614]: I! Detecting run_as_user...
Apr 22 00:21:02 sero1 start-amazon-cloudwatch-agent[7614]: 2022-04-21T22:21:02Z I! AWS_SDK_LOG_LEVEL is set to "LogDebug"
Apr 22 00:21:02 sero1 start-amazon-cloudwatch-agent[7614]: 2022-04-21T22:21:02Z I! CWAGENT_LOG_LEVEL is set to "DEBUG"
Apr 22 00:25:03 sero1 systemd[1]: amazon-cloudwatch-agent.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 00:25:03 sero1 systemd[1]: amazon-cloudwatch-agent.service: Failed with result 'exit-code'.

In logfile there are more details:

2022-04-21T22:21:02Z I! Starting AmazonCloudWatchAgent 1.247351.0-10-g2ece109-nightly-build
2022-04-21T22:21:02Z I! AWS SDK log level, LogDebug
2022-04-21T22:21:02Z I! Loaded inputs: net swap cpu disk diskio logfile mem
2022-04-21T22:21:02Z I! Loaded aggregators:
2022-04-21T22:21:02Z I! Loaded processors: delta ec2tagger
2022-04-21T22:21:02Z I! Loaded outputs: cloudwatch cloudwatchlogs
2022-04-21T22:21:02Z I! Tags enabled: host=sero1
2022-04-21T22:21:02Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"sero1", Flush Interval:1s
2022-04-21T22:21:02Z D! [agent] Initializing plugins
2022-04-21T22:21:02Z I! [logagent] starting
2022-04-21T22:21:02Z I! [logagent] found plugin cloudwatchlogs is a log backend
2022-04-21T22:21:02Z I! [logagent] found plugin logfile is a log collection
2022-04-21T22:21:32Z I! CWAGENT_LOG_LEVEL is set to "DEBUG"
2022-04-21T22:21:32Z I! AWS_SDK_LOG_LEVEL is set to "LogDebug"
2022-04-21T22:22:02Z D! Profiler dump:
[no stats is available...]
2022-04-21T22:23:02Z D! Profiler dump:
[no stats is available...]
2022-04-21T22:24:02Z D! Profiler dump:
[no stats is available...]
2022-04-21T22:25:02Z D! Profiler dump:
[no stats is available...]
2022-04-21T22:25:03Z E! [processors.ec2tagger] ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance
2022-04-21T22:25:03Z E! [telegraf] Error running agent: could not initialize processor ec2tagger: ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance
2022/04/22 00:26:09 I! 2022/04/22 00:26:03 D! [EC2] Found active network interface
2022/04/22 00:26:09 E! ec2metadata is not available
I! Detected the instance is OnPremise
2022/04/22 00:26:09 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.
2022/04/22 00:26:09 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json ...
2022/04/22 00:26:09 I! Valid Json input schema.
I! Detecting run_as_user...
Got Home directory: /root
I! Set home dir Linux: /root
I! SDKRegionWithCredsMap region:  eu-central-1
No csm configuration found.
Under path : /logs/ | Info : Got hostname sero1 as log_stream_name
Configuration validation first phase succeeded

2022/04/22 00:26:09 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2022/04/22 00:26:09 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2022/04/22 00:26:09 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json ...
2022/04/22 00:26:09 I! Valid Json input schema.
2022/04/22 00:26:09 I! Detected runAsUser: cwagent
2022/04/22 00:26:09 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var] to 996:996
2022/04/22 00:26:09 I! Set HOME: /home/cwagent

After this line service simply dies. No metrics in CloudWatch ever reported.

Answer 1

I had the same error, and in my case it was because I was setting "append_dimensions". (I was misreading the official document and setting custom field and value.)

This field can only contain values related to EC2 metadata, and it seems that if thisfield is configured, CloudWatch Agent tries to read processors.ec2tagger plugin and fails.

I could have uploaded my metrics and logs after deleting "append_dimensions" field.