Google Cloud: Health Check is not removing failed instance from UDP Internal Load Balancer

Question

I'm working in a project to move our SIP infra. to GCP.

I'm using an UDP Internal load balancer with a private IP to route calls from Asterisk to my Kamailio SBC, Asterisk is configured with the address IP of the load balancer as a single outgoing endpoint.

my Internal UDP Load Balancer operate on 5060 Frontend, a Backend with 2 SBC and basic http Health Check on port 80.

On each kamailio SBC I have my application listing on port 5060 and apache server on port 80 for health check so stopping httpd change the status of an instance to unhealthy.

forwarding-rules

# gcloud compute forwarding-rules describe ip-gateway-internal-lb-local-fontend --region=europe-west3
IPAddress: 10.156.0.15
IPProtocol: UDP
backendService: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3/backendServices/My-gateway-internal-lb-bservices
creationTimestamp: '2018-01-30T10:20:19.564-08:00'
description: ''
id: 'XXXXXXXXXXXXX'
kind: compute#forwardingRule
loadBalancingScheme: INTERNAL
name: ip-gateway-internal-lb-local-fontend
network: https://www.googleapis.com/compute/v1/projects/My-Project/global/networks/default
ports:
- '5060'
region: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3
selfLink: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3/forwardingRules/ip-gateway-internal-lb-local-fontend
subnetwork: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3/subnetworks/default

backend-service

# gcloud compute backend-services describe My-gateway-internal-lb-bservices --region=europe-west3
backends:
- balancingMode: CONNECTION
  description: ''
  group: https://www.googleapis.com/compute/v1/projects/My-Project/zones/europe-west3-a/instanceGroups/My-gateway-1xx
connectionDraining:
  drainingTimeoutSec: 0
creationTimestamp: '2018-01-30T10:15:10.688-08:00'
description: ''
fingerprint: XXXXXXXXX
healthChecks:
- https://www.googleapis.com/compute/v1/projects/My-Project/global/healthChecks/basic-check-internal-http
id: 'XXXXXXXXX'
kind: compute#backendService
loadBalancingScheme: INTERNAL
name: My-gateway-internal-lb-bservices
protocol: UDP
region: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3
selfLink: https://www.googleapis.com/compute/v1/projects/My-Project/regions/europe-west3/backendServices/My-gateway-internal-lb-bservices
sessionAffinity: NONE
timeoutSec: 3

health-check

# gcloud compute health-checks describe basic-check-internal-http
checkIntervalSec: 3
creationTimestamp: '2018-01-31T01:13:25.030-08:00'
description: ''
healthyThreshold: 2
httpHealthCheck:
  host: ''
  port: 80
  proxyHeader: NONE
  requestPath: /
id: 'XXXXXXXXXXXXXXXXXXXX'
kind: compute#healthCheck
name: basic-check-internal-http
selfLink: https://www.googleapis.com/compute/v1/projects/My-Project/global/healthChecks/basic-check-internal-http
timeoutSec: 3
type: HTTP
unhealthyThreshold: 2

All timeout are set to 3s, Internal UDP LB route rules done by the session affinity (the persistence) are not removed immediately, it takes about 15 min (without any traffic) to be removed.

Same case when an instance is healthy again, it takes 15 min to be considered by the LB and start receiving traffic.

I didn't had this problem when I was using an UDP load balancer with an External address IP, because my asterisk address sending the traffic are nated so the 5-tuple hash will be different for each call.

But with an UDP LB using an Internal IP the 5-tuple hash will be always the same (same src/dst IP:Port) so how I can configure the timeout of the session affinity (persistence) rules or force flushing the memory of my LB.

Maybe it's a Bug ! Has anyone run into the same problem? Thanks and looking forward if any someone can help me out with this issue ?

BR, Ouss

Answer 1

It is a bug on the Internal load balancer (UDP)

https://groups.google.com/forum/#!topic/gce-discussion/1uYmNoLgdGw

https://issuetracker.google.com/issues/72491707