Route53 DNS Aliased to Cloudfront distribution is resolving to Loopback address at some geographies

Question

I have a Route53 record (example.app) that is an Alias for an AWS Cloudfront Distribution that hosts my site.

I have numerous clients reporting that they are getting a timeout when attempting to access example.app in various geographies across the United States. There have been no reported issues in EMEA and APAC.

On an affected machine, using dig , I can see that I am able to resolve public IPs that point to AWS PoP servers (Domain name and IPs have been altered)

; <<>> DiG 9.10.6 <<>> example.app
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41637
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
www.example.app.       IN  A
;; ANSWER SECTION:
example.app.        60  IN  A   257.257.257.1
example.app.        60  IN  A   257.257.257.2
example.app.        60  IN  A   257.257.257.3
example.app.        60  IN  A   257.257.257.4
;; Query time: 88 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Fri Dec 09 09:45:39 EST 2022
;; MSG SIZE  rcvd: 122

This all looks healthy, however on attempting to visit example.com in my browser, the connection times out.

I can replicate the timeout with the following cURL.

curl 'https://example.app/' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' 
  -v

Which returns the following

 *    Trying 127.0.0.54:443
 * connect to 127.0.0.54 port 443 failed: Operation timed out
 * Failed to connect to example.app port 443 after 75031 ms: Operation timed out
 * Closing connection 0
curl: (28) Failed to connect to wirewax.app port 443 after 75031 ms: Operation timed out

Why is it attempting to connect to 127.0.0.54:443 given that's a loopback address? Any help as to what could be coming on would be much appreciated.

Answer 1

Turns out the issue was with the DNS of the ISP (Spectrum). Switching the DNS to google's (8.8.8.8) fixed the issue.