How to enable nginx reverse proxy to work with gRPC in .Net core?

Question

I am running into a problem where I am unable to get nginx to work properly with gRPC. I am using.Net core 3.1 to server an API that supports both REST and gRPC.

I am using below docker images:

• .Net Core 3.1 (aspnet:3.1-alpine)
• Nginx (nginx:latest)

Client is running locally as I'm just connecting via nginx to the docker container (port 8080 and 443 mapped to host)

I have built the API image in a docker container and am using docker compose to spin everything up.

My API is fairly straightforward when it comes to gRPC:

app.UseEndpoints(endpoints =>
{
   endpoints.MapGrpcService<CartService>();
   endpoints.MapControllers();
});

I have nginx as a reverse proxy in front of my API and below is my nginx config. But the rpc calls don't work. I can't connect to the gRPC service through a client and it returns a 502 request. I get a 2020/06/29 18:33:30 [error] 27#27: *3 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 172.20.0.1 . . After adding separate kestral endpoints (see my Edit1 below), I receive *1 upstream prematurely closed connection while reading response header from upstream when I look at Nginx logs.

The actual request is never even received by the server as nothing is logged server side when i peek into the docker logs.

There is little to no documentation on how to support gRPC through docker on.Net so unsure how to proceed. What needs to be configured/enabled further than what I have to get this working? Note that the REST part of the API works fine without issues. Unsure if SSL needs to be carried all the way up to the upstream servers (ie SSL at the API level at well).

The documentation I've seen on Nginx for gRPC has exactly what I have below. http_v2_module is enabled in Nginx and I can verify it works for the non gRPC part of the API through the response protocol.

http {
    upstream api {
        server apiserver:5001;
    }
    upstream function {
        server funcserver:5002;
    }

    # redirect all http requests to https
    server {
        listen 80 default_server;
        listen [::]:80 default_server;
        return 301 https://$host$request_uri;
    }
    server {
        server_name api.localhost;
        listen 443 http2 ssl ipv6only=on;
        ssl_certificate /etc/certs/api.crt;
        ssl_certificate_key /etc/certs/api.key;
        location /CartCheckoutService/ValidateCartCheckout {
            grpc_pass grpc://api;
            proxy_buffer_size          512k;
            proxy_buffers              4 256k;
            proxy_busy_buffers_size    512k;
            grpc_set_header Upgrade $http_upgrade;
            grpc_set_header Connection "Upgrade";
            grpc_set_header Connection keep-alive;
            grpc_set_header Host $host:$server_port;
            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            grpc_set_header X-Forwarded-Proto $scheme;
        }
        location / {
            proxy_pass http://api;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Connection keep-alive;
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_cache_bypass $http_upgrade;
        }
    }
    server {
        server_name func.localhost;
        listen 443 ssl;
        ssl_certificate /etc/certs/func.crt;
        ssl_certificate_key /etc/certs/func.key;
        location / {
            proxy_pass http://function;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection keep-alive;
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_cache_bypass $http_upgrade;
        }
    }
    gzip on;
    gzip_vary on;
    gzip_proxied no-cache no-store private expired auth;
    gzip_types text/plain text/css application/json application/xml;
}

---

Edit1: I've also tried to spin up separate endpoints for REST/gRPC. From this piece of documentation, when insecure requests come in, its automatically assumed to be Http1 requests. I configured kestrel manually to have 2 separate endpoints, two ports - one for http1+http2 and other for http2 requests.

services.Configure<KestrelServerOptions>(y =>
{
    y.ListenAnyIP(5010, o =>
    {
        o.Protocols = HttpProtocols.Http2;
        //o.UseHttps("./certs/backend.pfx", "password1");
    });

    y.ListenAnyIP(5001, o =>
    {
        o.Protocols = HttpProtocols.Http1AndHttp2;
    });
 });

In Nginx, I created a separate entries as well:

upstream api {
        server apiserver:5001;
    }
    upstream grpcservice {
        server apiserver:5010;
    }
    upstream function {
        server funcserver:5002;
    }

This does not work either. I even tried upstream SSL via making the htt2 endpoint accept only ssl connections but no dice.

---

Edit2

I have also tried below:

• Upstream SSL in Nginx - ie SSL between backend and reverse proxy
• Used Debian/Ubuntu based images instead of Alpine

None of them work either.

---

Edit 3: I was able to finally make this work:

location /CartCheckoutService/ValidateCartCheckout {
                grpc_pass grpc://api;
            }

For whatever reason, the only configuration for nginx that works is using grpc_pass only. It's not similar to proxy pass and the other configuration is not required. I am finally able to get this to work without having to do upstream SSL and just use the proxy like I meant to - terminate SSL at the proxy.

I'm still looking for a formal explanation otherwise I'll mark my own solution as the answer as I have tested it successfully.

Answer 1

Below is the solution that works:

location /CartCheckoutService/ValidateCartCheckout {
                grpc_pass grpc://api;
            }

The only configuration for nginx that works when using grpc is using grpc_pass only. It's not similar to proxy pass and the other configuration is not required (ie passing the headers/protocol/etc from the request). I am finally able to get this to work without having to do upstream SSL and just use the proxy like I meant to - terminate SSL at the proxy.

Answer 2

Nginx proxy_pass does not support http/2 . Since you're forwarding a gRPC connection, and gRPC requires http/2 the connection fails when it is sent via http/1 to the upstream.

Q: Will you support HTTP/2 on the upstream side as well, or only support HTTP/2 on the client side?

A: At the moment, we only support HTTP/2 on the client side. You can't configure HTTP/2 with proxy_pass.

https://www.nginx.com/blog/http2-module-nginx/#QandA

As you have found, you need to use grpc_pass to forward the connection to the upstream gRPC server.

Answer 3

I am using grpc server with https , In this case we need to change like below

grpc_pass grpcs://api;

I had a protocol configuration error in my routes and services, I should have changed grpc to grpcs , the problem has been solved, thanks a lot!