AWS CLB/ELB + HTTP/2 support

Question

The goal is to include HTTP/2 support in a simple stack : web app deployed in multiple EC2 instances + a transport-level CLB with the PROXY protocol policy enabled ( SSL:443 ➝ TCP:80 ) in order to offload SSL/TLS and to balance incoming HTTPS traffic.

Several reasons for the PROXY protocol : (1) execution of geolocation logic; (2) execution of simple access control rules; and (3) logging. All these features need access to a reliable (ie not trivially forgeable) client IP address. The only alternative to PROXY protocol in AWS would be switching to application-level balancing and using the XFF header to extract the remote IP address of the client. However that's not acceptable: anybody could trivially change its IP address simply injecting a fake XFF header in the incoming HTTPS request. AFAIK, AWS CLBs/ELBs do not inject a header containing the remote IP of the client (eg something like the True-Client-IP header in Akamai ).

Therefore, how to add H2 support to the stack? After some research all possible options look unsatisfactory:

• Current architecture is not valid because SSL/TLS is terminated in the CLB, but the CLB does not provide any options to announce H2 support through ALPN .

• An alternative using a CLB would be to stop using the SSL/TLS offloading feature and move it to EC2 instances (ie TCP:443 ➝ TCP:443 ). That way H2 support could be announced during the SSL/TLS handshake, but this option would require upgrading EC2 instances to support the additional SSL/TSL workload. Similar alternatives:

  • TCP:443 ➝ SSL:443 : similar to TCP:443 ➝ TCP:443 but allowing backend authentication using a list of trustworthy public key certificates.
  • SSL:443 ➝ SSL:443 : end-to-end encryption similar to TCP:443 ➝ SSL:443. Not really an option: (1) PROXY protocol is not supported for this combination (and using XFF is not an option too because this is transport-level balancing); and (2) client SSL/TLS handshake is executed in the CLB so H2 is not going to be announced.

• Other option would be replacing the CLB by an ELB ( HTTPS ➝ HTTP ). ELBs support H2. However (1) we'd need to rely on XFF to extract the client IP address (already explained why this is a problem); and (2) traffic between ELB and EC2 instances would be H1 (we'd like to let unencrypted H2 traffic reach EC2 instances). In other words, this is not an option.

To sum up, all options are problematic. IMHO the ideal solution would be to keep the original CLB (SSL:443 ➝ TCP:80; balancing + SSL/TLS offloading + PROXY protocol) and allow enabling a policy in the CLB to announce H2 support through ALPN. However I'm afraid this is not possible in AWS. Any alternatives to the CLB TCP:443 ➝ TCP:443 approach?

Answer 1

This answer doesn't provide a brilliant solution, because there may not be one, but I believe there's a gap in your understanding of AWS Application Load Balancer (ALB, also referred to as ELB/2.0).

anybody could trivially change its IP address simply injecting a fake XFF header in the incoming HTTPS request. AFAIK, AWS CLBs/ELBs do not inject a header containing the remote IP of the client

This is not correct on either count.

The remote IP of the client is the right-most IP address in X-Forwarded-For , as sent by the balancer to the instance. This cannot be spoofed. If the client includes one or more addresses in XFF, they are normalized by the balancer down to a single header, according to the rules for processing HTTP headers (left to right, first to last), and they appear to the left of the actual client IP.

Example request: I'm injecting two spoofed X-Forwarded-For headers into the request, one with 2 values, one with 1... here's what curl sends:

$ curl -v http://cx-xxxxxxxx-xxxx-xxxxxxxxxx.us-east-1.elb.amazonaws.com/test/dump/headers \
       -H 'X-Forwarded-For: 192.168.254.252, 10.10.10.10' \
       -H 'X-Forwarded-For: 172.16.16.16'
* Hostname was NOT found in DNS cache
*   Trying 52.x.x.x...
* Connected to cx-xxxxxxxx-xxxx-xxxxxxxxxx.us-east-1.elb.amazonaws.com (52.x.x.x) port 80 (#0)
> GET /test/dump/headers HTTP/1.1
> User-Agent: curl/7.35.0
> Host: cx-xxxxxxxx-xxxx-xxxxxxxxxx.us-east-1.elb.amazonaws.com
> Accept: */*
> X-Forwarded-For: 192.168.254.252, 10.10.10.10
> X-Forwarded-For: 172.16.16.16
>
< HTTP/1.1 200 OK

But here's what my instance sees:

GET /test/dump/headers HTTP/1.1
X-Forwarded-For: 192.168.254.252, 10.10.10.10, 172.16.16.16, 203.0.113.1
X-Forwarded-Proto: http
X-Forwarded-Port: 80
Host: cx-xxxxxxxx-xxxx-xxxxxxxxxx.us-east-1.elb.amazonaws.com
X-Amzn-Trace-Id: Root=1-58d0704b-557517de784f5exxxxxxxxxx
User-Agent: curl/7.35.0
Accept: */*

The headers I sent have been normalized by the balancer and the right-most value in X-Forwarded-For has been added by the balancer -- and it is the IP address of the machine where I'm running curl, that established the incoming connection.¹

That's how it always works with an ALB. If any XFF was provided by the client, those are not trusted, of course (although you should still log or save them, since they might be useful if the client was using a proxy that did correctly identify the client), but the last one on the right is always going to be the address of the external machine the established the connection to your ALB from the outside.

This is the way you always interpret X-Forwarded-For: -- from the bottom of the headers, up, and then right to left, because that's the way any properly behaving proxy (which ALB is, in this case) will handle them -- by appending its client's IP address (as reported from its network stack) to the right (or by adding an additional X-Forwarded-For header after any others -- either way is semantically valid, but ALB doesn't use the latter method). When parsing, when you encounter the first address that isn't one of your own, you stop -- that's the only address you can trust.

Additionally, if the client injects X-Forwarded-Proto: https when connecting using http -- trying to spoof having made a secure connection to the balancer when they aren't actually making one -- the ALB discards it. The instance sees only the truth: X-Forwarded-Proto: http .

---

Additionally, you might be overlooking the fact that while connecting to the instances with HTTP/1.1, the balancer does something else that's quite useful:

You can use HTTP/2 with HTTPS listeners. You can send up to 128 requests in parallel using one HTTP/2 connection. The load balancer converts these to individual HTTP/1.1 requests and distributes them across the healthy targets in the target group using the round robin routing algorithm.

http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-listeners.html

In my estimation, that capability takes the concept of load balancing to a whole new level.

---

¹ IP address of the the machine where I'm running curl. Astute observers will note that this is in fact an IP address from RFC-5737 , but it was reported to the instance as my IP address at home. I have not otherwise altered the request headers as seen by the instance except for sanitizing purposes. Otherwise, the original order and contents were precisely preserved, here.