[英]Ingress nginx stripping grpc request metadata headers
Problem Summary问题总结
When using ingress nginx controller to route requests to my grpc server, the request metadata headers get stripped out.当使用入口 nginx controller 将请求路由到我的 grpc 服务器时,请求元数据标头被删除。 When I directly hit the service the headers are present.
当我直接点击服务时,标题存在。
Ingress入口
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: <domain name>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: <service name>
port:
number: 50051
tls:
- hosts:
- <domain name>
secretName: <secret>
Troubleshooting I've done.我已经完成了故障排除。
example command I used to request to nginx.我曾经向 nginx 请求的示例命令。 When I inspect the server log, the "test.header" metadata is missing
当我检查服务器日志时,缺少“test.header”元数据
./grpcurl -vv -cacert <cert path> -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <domain name>:443 <endpoint>
example command to request directly to the service.直接向服务请求的示例命令。 The "test.header" metadata comes through
“test.header”元数据通过
./grpcurl -vv --plaintext -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <service name>:50051 <endpoint>
Inspecting nginx log with --v=5 gives me this.用 --v=5 检查 nginx 日志给了我这个。 No error message, but the header is not there.
没有错误消息,但 header 不存在。
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":method: POST"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":scheme: http"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":path: <path>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":authority: upstream_balancer"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-request-id: e89c7ad55ebd852a06416158be06f799"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-real-ip: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-for: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-host: <domain name>:443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-port: 443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-proto: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "te: trailers"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: 505249202a20485454502f322e300d0a0d0a534d0d0a0d0a00001204000000000000010000000000020000000000047fffffff0000040800000000007fff0000000136010400000001838644b16087aa534d6d83498f54482d9dcc42afdc2bde3d529a6b6c1a4c7ab716cee62158c454de3d529a6b6c1a4c7aaefacc5f5f418cb6b426c28e98a31d03a885b30089f2b585ed6950958d27972bcf91d1c8db658e479b10c0e341702dbd1940e4aebeff0087f2b5851d0b1abf87081765f5c4bbff008bf2b4a7b3c0ec90b22d29ec87081765f5c4bbff008cf2b4a7b3c0ec90b22d39d09f921d03498f51283db22e6a925b2a1d171a699f008cf2b4a7b3c0ec90b2..., len: 383
...
Looking through different documents, there seem to be some options for http翻阅不同的文档,http 似乎有一些选项
But I'm struggling to find anything for grpc.但我正在努力为 grpc 寻找任何东西。 The closest I could find was "grpc_set_header" flag in nginx configuration,
我能找到的最接近的是 nginx 配置中的“grpc_set_header”标志,
I'm very surprised and confused there isn't many documents or posts about what I'm experiencing, which makes me think that I'm doing something wrong.我感到非常惊讶和困惑,没有太多关于我所经历的文件或帖子,这让我觉得我做错了什么。 Any help would be appreciated.
任何帮助,将不胜感激。
Try with the in configmap use-forwarded-headers
尝试使用 in configmap
use-forwarded-headers
use-forwarded-headers: "true"
Doc ref: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers文档参考: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers
If your header container the underscore you can enable that also in configmap of Nginx ingress controller如果您的 header 容器下划线,您也可以在 Nginx 入口 controller 的配置映射中启用它
enable-underscores-in-headers: true
Update --更新 -
My issue was the dots in my header keys.我的问题是我的 header 键中的点。 https://trac.nginx.org/nginx/ticket/1558
https://trac.nginx.org/nginx/ticket/1558
Nginx was ignoring the keys with dots, and when I removed the dots the headers started coming through. Nginx 忽略了带点的键,当我删除点时,标题开始出现。 I also tried using "ignore_invalid_headers: false" config as suggested in the nginx ticket above, but that didn't work for me.
我还尝试按照上面的 nginx 票证中的建议使用“ignore_invalid_headers:false”配置,但这对我不起作用。 Just had to update the header keys.
只需更新 header 密钥。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.