why ingress nginx cannot proxy grpc when client using insecure?

Question

path: go-client --> ingress-nginx --> grpc pod

Because all the traffic is in our private network, so we didn't buy a public Certificate, rather we use a self-signed certificate. What happened is that the first code below worked well, but the second failed. I don't know why, and I want to know what the insecure exactly means.

code that worked well:

    cert, _ := credentials.NewClientTLSFromFile("./example.pem", "example.com")
    conn, err := grpc.DialContext(
        ctx,
        "example.com:443",
        grpc.WithTransportCredentials(cert),
        grpc.WithBlock(),
    )

code that received 400 bad request

    conn, err := grpc.DialContext(
        ctx,
        "example.com:443",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithBlock(),
    )

nginx access log for bad request

"PRI * HTTP/2.0" 400

ingress yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
  ingressClassName: nginx
  tls:
  - hosts: example.com
    secretName: example-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /foo/bar
        pathType: Prefix
        backend:
          service: grpc-svc
          port:
            name: grpc-port

Answer 1

Package insecure provides an implementation of the credentials.TransportCredentials interface which disables transport security. More specifically, it does not perform any TLS handshaking or use any certificates.

gRPC requires that the user pass it some credentials when attempting to create the ClientConn . If your deployment does not use any certificates and you know that it is secure (based on whatever reasons), then the insecure package will be your friend. But if you are using self-signed certificates, they are still certificates and a TLS handshake needs to happen here. So, in this case, you should continue using the code that you have mentioned at the top of your question. Hope this helps.