Nginx How to configure the Dynamic Proxy for AWS S3 bucket

Question

We have the following nginx configuration, corresponding to each s3 bucket,

location /bucketname1/ {
    proxy_pass https://bucketname1.s3.amazonaws.com/;
}

location /bucketname2/ {
    proxy_pass https://bucketname2.s3.amazonaws.com/;
}

location /bucketname3/ {
    proxy_pass https://bucketname3.s3.amazonaws.com/;
}

but as you can see, when more and more buckets are created, this location configurations are also increasing.

I plan to make a general location configuration to completely solve this problem.

But for this location:

location  ~* ^/([^/]+) {
    proxy_pass https://$1.s3.amazonaws.com/;
}

we encountered the error of "no resolver defined to resolve bucketname1", our nginx is deployed under EC2 Linux machine, does anyone know how I can find my resolver ip or is there a better way to solve this situation without using a resolver?

Thanks

Answer 1

This is a complex question. However:

A resolver normally exists for your VPC at IP equal to whatever its base CIDR range is except terminated with a 2.

For example, if your VPC was 10.0.0.0/8, you would find a resolver address at 10.0.0.2. There is also a resolver at 169.254.169.253.

If you ssh into a EC2 in the same VPC and run dig or nslookup commands, you will see it successfully resolving. Try:

dig @XXX2 <bucket-name>.s3.amazonaws.com

OR

dig @169.254.169.253 <bucket-name>.s3.amazonaws.com

Adding this to your nginx.conf should be helpful in resolving those IPs:

      resolver X.X.X.2 169.254.169.253;

This article explains the two default DNS resolver IP addresses that are created with your VPC:

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html

Here is helpful reading from the nginx docs on the server directive:

http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

These two stack overflow responses may also be very helpful in resolving your issue:

https://serverfault.com/questions/875150/resolver-directive-for-setting-up-nginx-reverse-proxy

AND

https://serverfault.com/questions/240476/how-to-force-nginx-to-resolve-dns-of-a-dynamic-hostname-everytime-when-doing-p