Istio envoy proxy issue

Question

We have configured the istio 1.4.0 with demo profile on Kubernetes cluster 1.15.1. It was working as expected but after some time facing issues with the application which are connecting to backend servers like mongo DB. Application pod is going in crashloopbackup and if i disabled istio it works properly.

Upon checking the istio-proxy logs found lines stating http/1.1 DPE and mongo DB IP and port number

Below is the Istio-proxy logs (sidecar),

#

 [2020-03-11T13:40:28.504Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49412 - - [2020-03-11T13:40:28.508Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:52062 - - [2020-03-11T13:40:28.528Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:37182 - - [2020-03-11T13:40:28.529Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - <mongo IP>:27017 10.233.92.103:49428 - - [2020-03-11T13:40:28.530Z] "- - HTTP/1.1" 0 DPE "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.26.61.18:27017 10.233.92.103:52078 - - [2020-03-11T13:40:28.569Z] "POST /intake/v2/events HTTP/1.1" 202 - "-" "-" 941 0 3 1 "-" "elasticapm-node/3.3.0 elastic-apm-http-client/9.3.0 node/10.12.0" "8954f0a1-709b-963c-a480-05b078955c89" "<apm>:8200" "10.26.61.45:8200" PassthroughCluster - <apm>:8200 10.233.92.103:49992 - - [2020-03-11T13:40:28.486Z] "- - -" 0 - "-" "-" 47 3671 98 - "-" "-" "-" "-" "<redis>:6379" PassthroughCluster 10.233.92.103:37254 <redis>:6379 10.233.92.103:37252 - - [2020-03-11T13:40:30.168Z] "- - -" 0 - "-" "-" 632 1212236 104 - "-" "-" "-" "-" "104.16.25.35:443" PassthroughCluster 10.233.92.103:60760 104.16.25.35:443 10.233.92.103:60758 - -```

#

and application logs giving error as below

{ err: 'socketHandler', trace: '', bin: undefined, parseState: { sizeOfMessage: 1347703880, bytesRead: undefined, stubBuffer: undefined } }

Answer 1

The issue has been resolved.

RCA :- I have manually created service and endpoint of MongoDB with port name as http

After that when i checked the listeners in proxy-config via istioctl command i found an entry with address 0.0.0.0 and port 27017

ADDRESS PORT TYPE 0.0.0.0 27017 TCP

In json output, i interpret that its going into blackholecluster even if i set allow_any in passthroughcluster.

The output in istio-proxy always give me DPE error.

After understanding the issue i changed the name from http to http1 and it worked properly.

Now need to understand why the name http was creating so much issue

Answer 2

Might be this issue: https://github.com/kubernetes/enhancements/issues/753

Basically, about ordering of containers. At startup time if your application container required networking and sidecar container(envoy proxy) not fully started , then application will raise an networking error.

Reference: https://discuss.istio.io/t/k8s-istio-sidecar-injection-with-other-init-containers/845