Getting 403 Forbidden from envoy when attempting to curl between sidecar enabled pods

Question

I'm using a Kubernetes/Istio setup and my list of pods and services are as below:

NAME                                                READY     STATUS    RESTARTS   AGE
hr--debug-deployment-86575cffb6-wl6rx               2/2       Running   0          33m
hr--hr-deployment-596946948d-jrd7g                  2/2       Running   0          33m

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
hr--debug-service                ClusterIP   10.104.160.61    <none>        80/TCP              33m
hr--hr-service                   ClusterIP   10.102.117.177   <none>        80/TCP              33m

I'm attempting to curl into hr--hr-service from hr--debug-deployment-86575cffb6-wl6rx

pasan@ubuntu:~/product-vick$ kubectl exec -it hr--debug-deployment-86575cffb6-wl6rx /bin/bash
Defaulting container name to debug.
Use 'kubectl describe pod/hr--debug-deployment-86575cffb6-wl6rx -n default' to see all of the containers in this pod.
root@hr--debug-deployment-86575cffb6-wl6rx:/# curl hr--hr-service -v
* Rebuilt URL to: hr--hr-service/
*   Trying 10.102.117.177...
* Connected to hr--hr-service (10.102.117.177) port 80 (#0)
> GET / HTTP/1.1
> Host: hr--hr-service
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< date: Thu, 03 Jan 2019 04:06:17 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host hr--hr-service left intact

Can you please explain why I'm getting a 403 forbidden by envoy and how I can troubleshoot it?

Answer 1

If you have the envoy sidecar injected it really depends on what type of authentication policy you have between your services. Are you using a MeshPolicy or a Policy ?

You can also try disabling authentication between your services to debug. Something like this (if your policy is defined like this):

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "hr--hr-service"
spec:
  targets:
  - name: hr--hr-service
  peers:
  - mTLS:
      mode: PERMISSIVE