AWS EC2 Windows 10 can't access metadata

Question

What could explain why an EC2 instance running Windows 10 does not consistently have access to its own metadata or userdata?

I know the userdata is set correctly because the exact same script was used for about thirty launches of t2.nano and c4.xlarge instances: the t2.nano never encountered any issue reading the metadata, but out of three attempts with a c4.xlarge only one had access to it. The script only differed by the name of instance (as per git history at least).

I followed the instructions below, and even from a Powershell, the Uri fails to load (cf. Figure 2).

Any hint is appreciated.

Answer 1

There is a script call InitializeInstance.ps1 that resets some configuration information.

For example, if the instance has changed subnets it might not work correctly due to cached routing rules. The InitializeInstance.ps1 can correct this.

Answer 2

We experienced the same issue on a Windows 2016 server on EC2. We noticed that the default gateway on the 169 IPs routes (persistent) where pointing at a non-existing (old?) gateway IP.

We changed the routes to the default gateway of the primary adapter, after that Instance Metadata started to work and AmazonSSMAgent service is running again.

Old situation:

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     172.16.3.129     172.16.3.152     15
     ....
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255     172.16.0.129      15
  169.254.169.250  255.255.255.255     172.16.0.129      15
  169.254.169.251  255.255.255.255     172.16.0.129      15

Notice the gateway on the persistent routes for 169 to point at an IP, which is not the default for 0.0.0.0. This 172.16.0.129 is also not pingable.

After changing the routes using route CHANGE:

route CHANGE 169.254.169.254 MASK 255.255.255.255  172.16.3.129 METRIC 15 IF 4 /P
route CHANGE 169.254.169.250 MASK 255.255.255.255  172.16.3.129 METRIC 15 IF 4 /P
route CHANGE 169.254.169.251 MASK 255.255.255.255  172.16.2.129 METRIC 15 IF 4 /P

Where:

• 172.16.3.129 is the default gateway on the primary network interface. This will be different on each instance.
• And 4 at then end METRIC 15 IF 4 is the interface ID of primary adapter, listed in interface list on route PRINT, this could also be different on each instance.

We now have:

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     172.16.3.129     172.16.3.152     15
....
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255     172.16.3.129      15
  169.254.169.250  255.255.255.255     172.16.3.129      15
  169.254.169.251  255.255.255.255     172.16.3.129      15
===========================================================================

This is basically what the for mentioned ProgramData/Amazon/EC2-Windows/Launch/Module/Scripts/Add-Routes.ps1 script does.

Answer 3

Can't edit John Rotenstein so I'll add it here,

This issue was solved at my end by initialise the instance again, it happen when you created an image in one subnet or vpc and launch it in a different one.

Warning this would change the admin password, make sure you have access to the required key to get the new password at console.

In order to initialise the instance only once run:

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule

in order to initialise it every boot run

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -SchedulePerBoot

the docs: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html

after running it, reboot the instance, get a new password from the console, rdp inside and you can get metadata and use aws s3 cli / powershell with the attached instance IAM role