Can't Curl Services running in the kubernetes cluster from the vm in istio mesh

Question

I am trying to deploy Istio on Virtual Machines. I am current architecture I have Kubernetes cluster which run the istio control plane (istiod) and a vm which is running the famous bookinfo istio application rating application. I am following the multi-network implementation as describe here ( https://istio.io/latest/docs/setup/install/virtual-machine/ ). I have followed each step of the documentation and have successfully completed all the setup.

Error: When I am trying to call the service running in kubernetes I get an error upstream connect error or disconnect/reset before headers. reset reason: connection failure upstream connect error or disconnect/reset before headers. reset reason: connection failure

I can successfully call the service running on the vm from kubernetes.

Log of istio services running on the vm

2022-09-02T14:24:08.165388Z info    FLAG: --domain=""
2022-09-02T14:24:08.165394Z info    FLAG: --help="false"
2022-09-02T14:24:08.165396Z info    FLAG: --log_as_json="false"
2022-09-02T14:24:08.165399Z info    FLAG: --log_caller=""
2022-09-02T14:24:08.165401Z info    FLAG: --log_output_level="dns:debug"
2022-09-02T14:24:08.165404Z info    FLAG: --log_rotate=""
2022-09-02T14:24:08.165407Z info    FLAG: --log_rotate_max_age="30"
2022-09-02T14:24:08.165409Z info    FLAG: --log_rotate_max_backups="1000"
2022-09-02T14:24:08.165412Z info    FLAG: --log_rotate_max_size="104857600"
2022-09-02T14:24:08.165414Z info    FLAG: --log_stacktrace_level="default:none"
2022-09-02T14:24:08.165420Z info    FLAG: --log_target="[stdout]"
2022-09-02T14:24:08.165423Z info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-09-02T14:24:08.165426Z info    FLAG: --outlierLogPath=""
2022-09-02T14:24:08.165428Z info    FLAG: --proxyComponentLogLevel=""
2022-09-02T14:24:08.165431Z info    FLAG: --proxyLogLevel="debug"
2022-09-02T14:24:08.165433Z info    FLAG: --serviceCluster="istio-proxy"
2022-09-02T14:24:08.165436Z info    FLAG: --stsPort="0"
2022-09-02T14:24:08.165438Z info    FLAG: --templateFile=""
2022-09-02T14:24:08.165441Z info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-09-02T14:24:08.165450Z info    FLAG: --vklog="0"
2022-09-02T14:24:08.165457Z info    Version 1.13.2-91533d04e894ff86b80acd6d7a4517b144f9e19a-Clean
2022-09-02T14:24:08.165587Z info    Proxy role  ips=[10.243.0.35 fe80::3cff:fe38:afc8] type=sidecar id=istio-on-vm-three.ratings domain=ratings.svc.cluster.local
2022-09-02T14:24:08.165626Z info    Apply mesh config from file defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  meshId: mesh1
  proxyMetadata:
    CANONICAL_REVISION: latest
    CANONICAL_SERVICE: ratings
    ISTIO_META_AUTO_REGISTER_GROUP: ratings
    ISTIO_META_CLUSTER_ID: cc90a48f0mfd7shso5g0
    ISTIO_META_DNS_CAPTURE: "true"
    ISTIO_META_MESH_ID: mesh1
    ISTIO_META_NETWORK: ""
    ISTIO_META_WORKLOAD_NAME: ratings
    ISTIO_METAJSON_LABELS: '{"app":"ratings","service.istio.io/canonical-name":"ratings","service.istio.io/canonical-revision":"latest"}'
    POD_NAMESPACE: ratings
    SERVICE_ACCOUNT: bookinfo-ratings
    TRUST_DOMAIN: cluster.local
  tracing:
    zipkin:
      address: zipkin.istio-system:9411

2022-09-02T14:24:08.166897Z info    Apply proxy config from env 
serviceCluster: ratings.ratings
controlPlaneAuthPolicy: MUTUAL_TLS

2022-09-02T14:24:08.167480Z info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
meshId: mesh1
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  CANONICAL_REVISION: latest
  CANONICAL_SERVICE: ratings
  ISTIO_META_AUTO_REGISTER_GROUP: ratings
  ISTIO_META_CLUSTER_ID: cc90a48f0mfd7shso5g0
  ISTIO_META_DNS_CAPTURE: "true"
  ISTIO_META_MESH_ID: mesh1
  ISTIO_META_NETWORK: ""
  ISTIO_META_WORKLOAD_NAME: ratings
  ISTIO_METAJSON_LABELS: '{"app":"ratings","service.istio.io/canonical-name":"ratings","service.istio.io/canonical-revision":"latest"}'
  POD_NAMESPACE: ratings
  SERVICE_ACCOUNT: bookinfo-ratings
  TRUST_DOMAIN: cluster.local
serviceCluster: ratings.ratings
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2022-09-02T14:24:08.167495Z info    JWT policy is third-party-jwt
2022-09-02T14:24:13.167597Z info    timed out waiting for platform detection, treating it as Unknown
2022-09-02T14:24:13.167892Z info    Opening status port 15020
2022-09-02T14:24:13.168029Z debug   dns initialized DNS search=[.] servers=[127.0.0.53:53]
2022-09-02T14:24:13.169553Z info    dns Starting local udp DNS server on 127.0.0.1:15053
2022-09-02T14:24:13.169584Z info    dns Starting local tcp DNS server on 127.0.0.1:15053
2022-09-02T14:24:13.169628Z info    CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2022-09-02T14:24:13.169647Z info    Using CA istiod.istio-system.svc:15012 cert with certs: /etc/certs/root-cert.pem
2022-09-02T14:24:13.169782Z info    citadelclient   Citadel client using custom root cert: /etc/certs/root-cert.pem
2022-09-02T14:24:13.182361Z info    ads All caches have been synced up in 5.02146778s, marking server ready
2022-09-02T14:24:13.182736Z info    sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-09-02T14:24:13.182795Z info    xdsproxy    Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "cc90a48f0mfd7shso5g0"
2022-09-02T14:24:13.182770Z info    sds Starting SDS grpc server
2022-09-02T14:24:13.183203Z info    starting Http service at 127.0.0.1:15004
2022-09-02T14:24:13.184810Z info    Pilot SAN: [istiod.istio-system.svc]
2022-09-02T14:24:13.186415Z info    Starting proxy agent
2022-09-02T14:24:13.186444Z info    Epoch 0 starting
2022-09-02T14:24:13.186463Z info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ  %l  envoy %n    %v -l debug --concurrency 2]
2022-09-02T14:24:13.264923Z info    xdsproxy    connected to upstream XDS server: istiod.istio-system.svc:15012
2022-09-02T14:24:13.284519Z info    cache   generated new workload certificate  latency=101.82115ms ttl=23h59m59.715492792s
2022-09-02T14:24:13.284554Z info    cache   Root cert has changed, start rotating root cert
2022-09-02T14:24:13.284578Z info    ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2022-09-02T14:24:13.284993Z info    cache   returned workload trust anchor from cache   ttl=23h59m59.715012276s
2022-09-02T14:24:13.327799Z info    ads ADS: new connection for node:istio-on-vm-three.ratings-1
2022-09-02T14:24:13.327908Z info    cache   returned workload certificate from cache    ttl=23h59m59.672096732s
2022-09-02T14:24:13.328260Z info    ads SDS: PUSH request for node:istio-on-vm-three.ratings resources:1 size:4.0kB resource:default
2022-09-02T14:24:13.367689Z info    ads ADS: new connection for node:istio-on-vm-three.ratings-2
2022-09-02T14:24:13.367769Z info    cache   returned workload trust anchor from cache   ttl=23h59m59.63223465s
2022-09-02T14:24:13.367948Z info    ads SDS: PUSH request for node:istio-on-vm-three.ratings resources:1 size:1.1kB resource:ROOTCA
2022-09-02T14:24:13.387123Z debug   dns updated lookup table with 96 hosts
2022-09-02T14:24:22.280792Z info    Agent draining Proxy
2022-09-02T14:24:22.280825Z info    Status server has successfully terminated
2022-09-02T14:24:22.281118Z error   accept tcp [::]:15020: use of closed network connection
2022-09-02T14:24:22.282028Z info    Graceful termination period is 5s, starting...
2022-09-02T14:24:27.282551Z info    Graceful termination period complete, terminating remaining proxies.
2022-09-02T14:24:27.282610Z warn    Aborted all epochs
2022-09-02T14:24:27.282622Z warn    Aborting epoch 0
2022-09-02T14:24:27.282889Z info    Epoch 0 aborted normally
2022-09-02T14:24:27.282899Z info    Agent has successfully terminated
2022-09-02T14:24:57.386419Z info    FLAG: --concurrency="0"
2022-09-02T14:24:57.386463Z info    FLAG: --domain=""
2022-09-02T14:24:57.386471Z info    FLAG: --help="false"
2022-09-02T14:24:57.386474Z info    FLAG: --log_as_json="false"
2022-09-02T14:24:57.386477Z info    FLAG: --log_caller=""
2022-09-02T14:24:57.386480Z info    FLAG: --log_output_level="dns:debug"
2022-09-02T14:24:57.386482Z info    FLAG: --log_rotate=""
2022-09-02T14:24:57.386486Z info    FLAG: --log_rotate_max_age="30"
2022-09-02T14:24:57.386489Z info    FLAG: --log_rotate_max_backups="1000"
2022-09-02T14:24:57.386492Z info    FLAG: --log_rotate_max_size="104857600"
2022-09-02T14:24:57.386495Z info    FLAG: --log_stacktrace_level="default:none"
2022-09-02T14:24:57.386504Z info    FLAG: --log_target="[stdout]"
2022-09-02T14:24:57.386507Z info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-09-02T14:24:57.386510Z info    FLAG: --outlierLogPath=""
2022-09-02T14:24:57.386512Z info    FLAG: --proxyComponentLogLevel=""
2022-09-02T14:24:57.386515Z info    FLAG: --proxyLogLevel="debug"
2022-09-02T14:24:57.386518Z info    FLAG: --serviceCluster="istio-proxy"
2022-09-02T14:24:57.386521Z info    FLAG: --stsPort="0"
2022-09-02T14:24:57.386533Z info    FLAG: --templateFile=""
2022-09-02T14:24:57.386544Z info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2022-09-02T14:24:57.386553Z info    FLAG: --vklog="0"
2022-09-02T14:24:57.386559Z info    Version 1.13.2-91533d04e894ff86b80acd6d7a4517b144f9e19a-Clean
2022-09-02T14:24:57.386705Z info    Proxy role  ips=[10.243.0.35 fe80::3cff:fe38:afc8] type=sidecar id=istio-on-vm-three.ratings domain=ratings.svc.cluster.local
2022-09-02T14:24:57.386749Z info    Apply mesh config from file defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  meshId: mesh1
  proxyMetadata:
    CANONICAL_REVISION: latest
    CANONICAL_SERVICE: ratings
    ISTIO_META_AUTO_REGISTER_GROUP: ratings
    ISTIO_META_CLUSTER_ID: cc90a48f0mfd7shso5g0
    ISTIO_META_DNS_CAPTURE: "true"
    ISTIO_META_MESH_ID: mesh1
    ISTIO_META_NETWORK: ""
    ISTIO_META_WORKLOAD_NAME: ratings
    ISTIO_METAJSON_LABELS: '{"app":"ratings","service.istio.io/canonical-name":"ratings","service.istio.io/canonical-revision":"latest"}'
    POD_NAMESPACE: ratings
    SERVICE_ACCOUNT: bookinfo-ratings
    TRUST_DOMAIN: cluster.local
  tracing:
    zipkin:
      address: zipkin.istio-system:9411

2022-09-02T14:24:57.387852Z info    Apply proxy config from env 
serviceCluster: ratings.ratings
controlPlaneAuthPolicy: MUTUAL_TLS

2022-09-02T14:24:57.388363Z info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
meshId: mesh1
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  CANONICAL_REVISION: latest
  CANONICAL_SERVICE: ratings
  ISTIO_META_AUTO_REGISTER_GROUP: ratings
  ISTIO_META_CLUSTER_ID: cc90a48f0mfd7shso5g0
  ISTIO_META_DNS_CAPTURE: "true"
  ISTIO_META_MESH_ID: mesh1
  ISTIO_META_NETWORK: ""
  ISTIO_META_WORKLOAD_NAME: ratings
  ISTIO_METAJSON_LABELS: '{"app":"ratings","service.istio.io/canonical-name":"ratings","service.istio.io/canonical-revision":"latest"}'
  POD_NAMESPACE: ratings
  SERVICE_ACCOUNT: bookinfo-ratings
  TRUST_DOMAIN: cluster.local
serviceCluster: ratings.ratings
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2022-09-02T14:24:57.388378Z info    JWT policy is third-party-jwt
2022-09-02T14:25:02.388947Z info    timed out waiting for platform detection, treating it as Unknown
2022-09-02T14:25:02.389180Z debug   dns initialized DNS search=[.] servers=[127.0.0.53:53]
2022-09-02T14:25:02.389248Z info    dns Starting local udp DNS server on 127.0.0.1:15053
2022-09-02T14:25:02.389249Z info    Opening status port 15020
2022-09-02T14:25:02.389413Z info    dns Starting local tcp DNS server on 127.0.0.1:15053
2022-09-02T14:25:02.389432Z info    CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2022-09-02T14:25:02.389445Z info    Using CA istiod.istio-system.svc:15012 cert with certs: /etc/certs/root-cert.pem
2022-09-02T14:25:02.389532Z info    citadelclient   Citadel client using custom root cert: /etc/certs/root-cert.pem
2022-09-02T14:25:02.402154Z info    ads All caches have been synced up in 5.019952409s, marking server ready
2022-09-02T14:25:02.402449Z info    sds SDS server for workload certificates started, listening on "etc/istio/proxy/SDS"
2022-09-02T14:25:02.402475Z info    xdsproxy    Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "cc90a48f0mfd7shso5g0"
2022-09-02T14:25:02.402487Z info    sds Starting SDS grpc server
2022-09-02T14:25:02.402794Z info    starting Http service at 127.0.0.1:15004
2022-09-02T14:25:02.403926Z info    Pilot SAN: [istiod.istio-system.svc]
2022-09-02T14:25:02.405489Z info    Starting proxy agent
2022-09-02T14:25:02.405522Z info    Epoch 0 starting
2022-09-02T14:25:02.405560Z info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ  %l  envoy %n    %v -l debug --concurrency 2]
2022-09-02T14:25:02.480867Z info    xdsproxy    connected to upstream XDS server: istiod.istio-system.svc:15012
2022-09-02T14:25:02.552937Z info    ads ADS: new connection for node:istio-on-vm-three.ratings-1
2022-09-02T14:25:02.592884Z info    ads ADS: new connection for node:istio-on-vm-three.ratings-2
2022-09-02T14:25:02.602362Z info    cache   generated new workload certificate  latency=199.854356ms ttl=23h59m59.397649371s
2022-09-02T14:25:02.602401Z info    cache   Root cert has changed, start rotating root cert
2022-09-02T14:25:02.602421Z info    ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2022-09-02T14:25:02.602531Z info    cache   returned workload trust anchor from cache   ttl=23h59m59.397477611s
2022-09-02T14:25:02.602586Z info    cache   returned workload certificate from cache    ttl=23h59m59.397417006s
2022-09-02T14:25:02.602881Z info    cache   returned workload trust anchor from cache   ttl=23h59m59.397122534s
2022-09-02T14:25:02.604303Z info    ads SDS: PUSH request for node:istio-on-vm-three.ratings resources:1 size:1.1kB resource:ROOTCA
2022-09-02T14:25:02.604361Z info    cache   returned workload trust anchor from cache   ttl=23h59m59.395642519s
2022-09-02T14:25:02.604393Z info    ads SDS: PUSH for node:istio-on-vm-three.ratings resources:1 size:1.1kB resource:ROOTCA
2022-09-02T14:25:02.604384Z info    ads SDS: PUSH request for node:istio-on-vm-three.ratings resources:1 size:4.0kB resource:default
2022-09-02T14:25:02.622631Z debug   dns updated lookup table with 96 hosts
2022-09-02T14:25:04.329218Z debug   dns request ;; opcode: QUERY, status: NOERROR, id: 24280
;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;details.default.svc.   IN   AAAA

;; ADDITIONAL SECTION:

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
    protocol=udp edns=true id=6240baac-c243-45be-9a10-dfe500a83cfa
2022-09-02T14:25:04.329282Z debug   dns response for hostname "details.default.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 24280
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;details.default.svc.   IN   AAAA
    protocol=udp edns=true id=6240baac-c243-45be-9a10-dfe500a83cfa
2022-09-02T14:25:04.329305Z debug   dns request ;; opcode: QUERY, status: NOERROR, id: 17619
;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;details.default.svc.   IN   A

;; ADDITIONAL SECTION:

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
    protocol=udp edns=true id=30fd3d3c-efed-4a27-b8ba-113f56efb67d
2022-09-02T14:25:04.329371Z debug   dns response for hostname "details.default.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 17619
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;details.default.svc.   IN   A

;; ANSWER SECTION:
details.default.svc.    30  IN  A   172.21.199.92
    protocol=udp edns=true id=30fd3d3c-efed-4a27-b8ba-113f56efb67d

Gateway configuration for istiod

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"istiod-gateway","namespace":"istio-system"},"spec":{"selector":{"istio":"eastwestgateway"},"servers":[{"hosts":["*"],"port":{"name":"tls-istiod","number":15012,"protocol":"tls"},"tls":{"mode":"PASSTHROUGH"}},{"hosts":["*"],"port":{"name":"tls-istiodwebhook","number":15017,"protocol":"tls"},"tls":{"mode":"PASSTHROUGH"}}]}}
  creationTimestamp: '2022-09-02T13:54:17Z'
  generation: 1
  managedFields:
    - apiVersion: networking.istio.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:selector:
            .: {}
            f:istio: {}
          f:servers: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2022-09-02T13:54:17Z'
  name: istiod-gateway
  namespace: istio-system
  resourceVersion: '3685'
  uid: 23f776c9-a4d1-43a7-8992-72be4f933d9d
spec:
  selector:
    istio: eastwestgateway
  servers:
    - hosts:
        - '*'
      port:
        name: tls-istiod
        number: 15012
        protocol: tls
      tls:
        mode: PASSTHROUGH
    - hosts:
        - '*'
      port:
        name: tls-istiodwebhook
        number: 15017
        protocol: tls
      tls:
        mode: PASSTHROUGH

Virtual service for istiod

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"istiod-vs","namespace":"istio-system"},"spec":{"gateways":["istiod-gateway"],"hosts":["*"],"tls":[{"match":[{"port":15012,"sniHosts":["*"]}],"route":[{"destination":{"host":"istiod.istio-system.svc.cluster.local","port":{"number":15012}}}]},{"match":[{"port":15017,"sniHosts":["*"]}],"route":[{"destination":{"host":"istiod.istio-system.svc.cluster.local","port":{"number":443}}}]}]}}
  creationTimestamp: '2022-09-02T13:54:17Z'
  generation: 1
  managedFields:
    - apiVersion: networking.istio.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:gateways: {}
          f:hosts: {}
          f:tls: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2022-09-02T13:54:17Z'
  name: istiod-vs
  namespace: istio-system
  resourceVersion: '3686'
  uid: d1b88fde-20a3-48dd-a549-dfe77407e206
spec:
  gateways:
    - istiod-gateway
  hosts:
    - '*'
  tls:
    - match:
        - port: 15012
          sniHosts:
            - '*'
      route:
        - destination:
            host: istiod.istio-system.svc.cluster.local
            port:
              number: 15012
    - match:
        - port: 15017
          sniHosts:
            - '*'
      route:
        - destination:
            host: istiod.istio-system.svc.cluster.local
            port:
              number: 443

Please let me know if you need more information to debug/

Answer 1

After a lot of debugging and trial and error I found the problem and solved. First the variables in definition to create workload group in the official istio documentation is not explained properly. As per the official documentation in the workload group we need to mention the network of the vm but doesn't which network as a vm can have interfaces mapping to a public and private network. The solution is that you need to mention the network ip which is mapping to default network interface ie in my case my eth0 interface mapped to the private ip of the vm, hence for me the workload definition was something like this

apiVersion: networking.istio.io/v1alpha3
kind: WorkloadGroup
metadata:
  name: "${VM_APP}"
  namespace: "${VM_NAMESPACE}"
spec:
  metadata:
    labels:
      app: "${VM_APP}"
  template:
    serviceAccount: "${SERVICE_ACCOUNT}"
    network: "${VM'S_PRIVATE_IP}"
  probe:
    periodSeconds: 5
    initialDelaySeconds: 1
    httpGet:
      port: 8080
      path: /ready

Second the command provided in the docs to create the workload entry is incomplete. To get a mesh expansion to work in a multi-network mesh the command should be

istioctl x workload entry configure -f workloadgroup.yaml -o "${WORK_DIR}" --clusterID "${CLUSTER}" --ingressIP ${EAST_WEST_GATEWAY_IP_ADDRESS} --externalIP ${PRIVATE_IP_OF_THE_VM or ETH0_IP_ADDRESS} --autoregister