Keycloak Kubernetes GKE NGINX Ingress - Session get lost after pod restart on page reload and returns 502 Bad Gateway

Question

I have setup a Keycloak Cluster in GKE with NGINX as Ingress Controller. I have use the Codecentrics Helm Chart: [https://github.com/codecentric/helm-charts/tree/master/charts/keycloak][Keycloak Helm Chart]

I am using JDBC_PING for JGroups and have the following cli script and Ingress config. I have replicas set to 2. When I kill a pod the session is still usable and everything is working fine, I can navigate in the keycloak admin interface and do everything. But when I hit F5 to reload the page I receive an 502 Bad Gateway error. Sometimes it does recover and I can just reload and everything is just fine, but sometimes I have to delete the cookies completely to make it work again.

I am not sure where the issue is coming from.

Cookies in Browser:

MySQL Table JGROUPSPING:

Ingress Annotations:

  annotations: 
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/limit-rate: "150"
    nginx.ingress.kubernetes.io/limit-rps: "150"
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "21600"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "21600"
    nginx.ingress.kubernetes.io/server-snippet: |
      location /auth/realms/master/metrics {
          return 403;
      } 

extra envs:

# Additional environment variables for Keycloak
extraEnv: |
  - name: KEYCLOAK_STATISTICS
    value: all
  - name: PROXY_ADDRESS_FORWARDING
    value: "true"
  - name: KEYCLOAK_USER
    value: '{{ .Values.ADMIN_USER }}'
  - name: KEYCLOAK_PASSWORD
    value: '{{ .Values.ADMIN_PASS }}'
  - name: JAVA_OPTS
    value: >-
      -XX:+UseContainerSupport
      -XX:MaxRAMPercentage=50.0
      -Djava.net.preferIPv4Stack=true
      -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS
      -Djava.awt.headless=true
  - name: JGROUPS_DISCOVERY_PROTOCOL
    value: JDBC_PING  
  - name: CACHE_OWNERS_COUNT
    value: "2"
  - name: CACHE_OWNERS_AUTH_SESSIONS_COUNT
    value: "2"
  - name: DB_VENDOR
    value: mysql
  - name: DB_ADDR
    value: "127.0.0.1"
  - name: DB_PORT
    value: "3306"
  - name: DB_DATABASE
    value: keycloak_prod
  - name: DB_USER
    value: '{{ .Values.SQL_USER }}'
  - name: DB_PASSWORD
    value: '{{ .Values.SQL_PASS }}'

Keycloak CLI script:

embed-server --server-config=standalone-ha.xml --std-out=echo
batch


echo Configuring node identifier

## Sets the node identifier to the node name (= pod name). Node identifiers have to be unique. They can have a
## maximum length of 23 characters. Thus, the chart's fullname template truncates its length accordingly.
/subsystem=transactions:write-attribute(name=node-identifier, value=${jboss.node.name})
echo NodeName: ${jboss.node.name}
echo Finished configuring node identifier

echo CUSTOM_CONFIG: executing CONFIG FOR K8S Failover Support


echo "------------------------------------------------------------------------------------------------------------"
echo "---------------------------------CUSTOM STARTUP CONFIG------------------------------------------------------"
echo "------------------------------------------------------------------------------------------------------------"

## JDBC PING

/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:write-attribute(name=owners, value=${env.CACHE_OWNERS_COUNT:2})

/subsystem=jgroups/stack=tcp:remove()
/subsystem=jgroups/stack=tcp:add()
/subsystem=jgroups/stack=tcp/transport=TCP:add(socket-binding="jgroups-tcp")
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING:add()
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=datasource_jndi_name:add(value=java:jboss/datasources/KeycloakDS)

/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=initialize_sql:add(value="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, ping_data varbinary(5000) DEFAULT NULL, PRIMARY KEY (own_addr, cluster_name)) ENGINE=InnoDB DEFAULT CHARSET=utf8")

/subsystem=jgroups/stack=tcp/protocol=MERGE3:add()
/subsystem=jgroups/stack=tcp/protocol=FD_SOCK:add(socket-binding="jgroups-tcp-fd")
/subsystem=jgroups/stack=tcp/protocol=FD:add()
/subsystem=jgroups/stack=tcp/protocol=VERIFY_SUSPECT:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.NAKACK2:add()
/subsystem=jgroups/stack=tcp/protocol=UNICAST3:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.STABLE:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS:add()
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS/property=max_join_attempts:add(value=5)
/subsystem=jgroups/stack=tcp/protocol=MFC:add()
/subsystem=jgroups/stack=tcp/protocol=FRAG3:add()

/subsystem=jgroups/stack=udp:remove()
/subsystem=jgroups/channel=ee:write-attribute(name=stack, value=tcp)
/socket-binding-group=standard-sockets/socket-binding=jgroups-mping:remove()


## Cache Setup for Failover
/subsystem=infinispan/cache-container=keycloak/distributed-cache=sessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=authenticationSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=clientSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=offlineClientSessions:remove()
/subsystem=infinispan/cache-container=keycloak/distributed-cache=loginFailures:remove()

/subsystem=infinispan/cache-container=keycloak/replicated-cache=sessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=authenticationSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=offlineSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=clientSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=offlineClientSessions:add()
/subsystem=infinispan/cache-container=keycloak/replicated-cache=loginFailures:add()

echo "------------------------------------------------------------------------------------------------------------"
echo "---------------------------------CUSTOM STARTUP CONFIG DONE!------------------------------------------------"
echo "------------------------------------------------------------------------------------------------------------"

run-batch

try
    :resolve-expression(expression=${env.JGROUPS_DISCOVERY_EXTERNAL_IP})
    /subsystem=jgroups/stack=tcp/transport=TCP/property=external_addr/:add(value=${env.JGROUPS_DISCOVERY_EXTERNAL_IP})
catch
    echo "JGROUPS_DISCOVERY_EXTERNAL_IP maybe not set."
end-try

stop-embedded-server

Log of the restarted Pod: log-restarted-pod.txt

Log of the still running pod: log-still-running-pod.txt

Answer 1

I managed to figure out this issue, we need to add below annotation to our ingress.yaml file.

nginx.ingress.kubernetes.io/proxy-buffer-size: "12k"