How do I forward headers to different services in Kubernetes (Istio)

Question

I have a sample application (web-app, backend-1, backend-2) deployed on minikube all under a JWT policy, and they all have proper destination rules, Istio sidecar and MTLS enabled in order to secure the east-west traffic.

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: oidc
spec:
  targets:
  - name: web-app
  - name: backend-1
  - name: backend-2
  peers:
  - mtls: {}
  origins:
  - jwt:
      issuer: "http://myurl/auth/realms/test"
      jwksUri: "http://myurl/auth/realms/test/protocol/openid-connect/certs"
  principalBinding: USE_ORIGIN

When I run the following command I receive a 401 unauthorized response when requesting the data from the backend, which is due to $TOKEN not being forwarded to backend-1 and backend-2 headers during the http request.

$> curl http://minikubeip/api "Authorization: Bearer $TOKEN"

Is there a way to forward http headers to backend-1 and backend-2 using native kubernetes/istio? Am I forced to make application code changes to accomplish this?

Edit: This is the error I get after applying my oidc policy. When I curl web-app with the auth token I get

{"errors":[{"code":"APP_ERROR_CODE","message":"401 Unauthorized"}

Note that when I curl backend-1 or backend-2 with the same auth-token I get the appropriate data. Also, there is no other destination rule/policy applied to these services currently, policy enforcement is on, and my istio version is 1.1.15. This is the policy I am applying:

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: default
  namespace: default
spec:
  # peers:
  # - mtls: {}
  origins:
  - jwt:
      issuer: "http://10.148.199.140:8080/auth/realms/test"
      jwksUri: "http://10.148.199.140:8080/auth/realms/test/protocol/openid-connect/certs"
  principalBinding: USE_ORIGIN

Answer 1

should the token be propagated to backend-1 and backend-2 without any other changes?

Yes, policy should transfer token to both backend-1 and backend-2

There is a github issue , where users had same issue like You

A few informations from there:

The JWT is verified by an Envoy filter, so you'll have to check the Envoy logs. For the code, see https://github.com/istio/proxy/tree/master/src/envoy/http/jwt_auth

Pilot retrieves the JWKS to be used by the filter (it is inlined into the Envoy config), you can find the code for that in pilot/pkg/security

And another problem with that in stackoverflow

where accepted answer is:

The problem was resolved with two options: 1. Replace Service Name and port by external server ip and external port (for issuer and jwksUri) 2. Disable the usage of mTLS and its policy (Known issue: https://github.com/istio/istio/issues/10062 ).

From istio documentation

For each service, Istio applies the narrowest matching policy. The order is: service-specific > namespace-wide > mesh-wide. If more than one service-specific policy matches a service, Istio selects one of them at random. Operators must avoid such conflicts when configuring their policies.

To enforce uniqueness for mesh-wide and namespace-wide policies, Istio accepts only one authentication policy per mesh and one authentication policy per namespace. Istio also requires mesh-wide and namespace-wide policies to have the specific name default.

If a service has no matching policies, both transport authentication and origin authentication are disabled.

Answer 2

Istio supports header propagation. Probably didn't support when this thread was created.

You can allow the original header to be forwarded by using forwardOriginalToken : true in JWTRules or forward a valid JWT payload using outputPayloadToHeader in JWTRules .

Reference: ISTIO JWTRule documentation