Shiro Session Clustering with Hazelcast, not on the Web

Question

I have written a Netty-based REST service which is deployed directly (ie without a Web/Application server). For authn, authz and session management I am using Apache Shiro. Now I want to be able to load-balance this service without using sticky sessions on our load-balancer; and I am trying to set that up using in-process Hazelcast instances.

So far I have managed to have two instances of my service running on two different machines create a Hazelcast cluster with two members; but even so, if I authenticate on one machine, the other still doesn't recognize the session. So I think I have probably made a mistake in my shiro.ini and/or my hazelcast.xml configuration files.

I am essentially using the HazelCastCacheManager class from here https://github.com/stormpath/shiro-hazelcast-web-sample/blob/master/src/main/java/com/stormpath/samples/shiro/hazelcast/cache/HazelcastCacheManager.java

Here is my hazelcast.xml (our setup doesn't allow multicast, I have to specify the machine IPs directly):

<hazelcast>
  <properties>
    <property name="hazelcast.logging.type">slf4j</property>
  </properties>

  <map name="shiro-activeSessionsCache">
    <async-backup-count>1</async-backup-count>
    <time-to-live-seconds>600</time-to-live-seconds>
  </map>

  <network>
    <join>
      <multicast enabled="false"></multicast>
      <tcp-ip enabled="true">
        <member>x.x.x.x:8050</member> <!-- server A -->
        <member>x.x.x.x:8050</member> <!-- server B -->
      </tcp-ip>
    </join>
    <interfaces enabled="true">
        <interface>x.x.x.*</interface>
    </interfaces>
    <port port-count="20" auto-increment="false">8050</port>    
  </network>

</hazelcast>

And a portion of my shiro.ini

sessionDAO = org.apache.shiro.session.mgt.eis.EnterpriseCacheSessionDAO
sessionManager = org.apache.shiro.session.mgt.DefaultSessionManager

sessionDAO.activeSessionsCacheName = shiro-activeSessionsCache
securityManager.sessionManager.sessionDAO = $sessionDAO

cacheManager = HazelcastCacheManager

securityManager.cacheManager = $cacheManager

sessionManager.globalSessionTimeout = 600000
sessionValidationScheduler = org.apache.shiro.session.mgt.ExecutorServiceSessionValidationScheduler
sessionValidationScheduler.interval = 600000
sessionManager.sessionValidationScheduler = $sessionValidationScheduler

securityManager.sessionManager=$sessionManager

What am I missing? How can I instruct Shiro to share the session among all Hazelcast instances? Or is my mistake in the HazelcastCacheManager class?

Here is the log Hazelcast produces when I start up my two services:

This is the first service starting up:

log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 24, 2015 9:03:42 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Interfaces is enabled, trying to pick one address matching to one of: [x.x.x.*]
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Prefer IPv4 stack is true.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Picked Address[x.x.x.x]:8050, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=8050], bind any local is true
[main] INFO com.hazelcast.spi.OperationService - [x.x.x.x]:8050 [dev] [3.5.3] Backpressure is disabled
[main] INFO com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor - [x.x.x.x]:8050 [dev] [3.5.3] Starting with 16 generic operation threads and 32 partition operation threads.
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Hazelcast 3.5.3 (20151011 - 64c663a) starting at Address[x.x.x.x]:8050
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
[main] INFO com.hazelcast.instance.Node - [x.x.x.x]:8050 [dev] [3.5.3] Creating TcpIpJoiner
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTING
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Connecting to /x.x.x.x:8050, timeout: 0, bind-any: true
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Could not connect to: /x.x.x.x:8050. Reason: SocketException[Connection refused to address /x.x.x.x:8050]
[cached1] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is added to the blacklist.
[main] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [x.x.x.x]:8050 [dev] [3.5.3]

Members [1] {
        Member [x.x.x.x]:8050 this
}

And this is the continuation when the second service starts:

[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTED
[main] INFO org.apache.shiro.config.IniSecurityManagerFactory - Realms have been explicitly set on the SecurityManager instance - auto-setting of realms will not occur.
Open your web browser and navigate to https://127.0.0.1:8113/
[hz._hzInstance_1_dev.IO.thread-Acceptor] INFO com.hazelcast.nio.tcp.SocketAcceptor - [x.x.x.x]:8050 [dev] [3.5.3] Accepting socket connection from /x.x.x.x:35316
[cached2] INFO com.hazelcast.nio.tcp.TcpIpConnectionManager - [x.x.x.x]:8050 [dev] [3.5.3] Established socket connection between /x.x.x.x:8050
[hz._hzInstance_1_dev.generic-operation.thread-10] INFO com.hazelcast.cluster.ClusterService - [x.x.x.x]:8050 [dev] [3.5.3]

Members [2] {
        Member [x.x.x.x]:8050 this
        Member [x.x.x.x]:8050
}

This is the second service starting up, with the first one already running:

log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 24, 2015 9:04:16 AM com.hazelcast.config.XmlConfigLocator
INFO: Loading 'hazelcast.xml' from working directory.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Interfaces is enabled, trying to pick one address matching to one of: [x.x.x.*]
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Prefer IPv4 stack is true.
[main] INFO com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [dev] [3.5.3] Picked Address[x.x.x.x]:8050, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=8050], bind any local is true
[main] INFO com.hazelcast.spi.OperationService - [x.x.x.x]:8050 [dev] [3.5.3] Backpressure is disabled
[main] INFO com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor - [x.x.x.x]:8050 [dev] [3.5.3] Starting with 8 generic operation threads and 16 partition operation threads.
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Hazelcast 3.5.3 (20151011 - 64c663a) starting at Address[x.x.x.x]:8050
[main] INFO com.hazelcast.system - [x.x.x.x]:8050 [dev] [3.5.3] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
[main] INFO com.hazelcast.instance.Node - [x.x.x.x]:8050 [dev] [3.5.3] Creating TcpIpJoiner
[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTING
[cached1] INFO com.hazelcast.nio.tcp.SocketConnector - [x.x.x.x]:8050 [dev] [3.5.3] Connecting to /x.x.x.x:8050, timeout: 0, bind-any: true
[cached1] INFO com.hazelcast.nio.tcp.TcpIpConnectionManager - [x.x.x.x]:8050 [dev] [3.5.3] Established socket connection between /x.x.x.x:35316
[hz._hzInstance_1_dev.generic-operation.thread-3] INFO com.hazelcast.cluster.ClusterService - [x.x.x.x]:8050 [dev] [3.5.3]

Members [2] {
        Member [x.x.x.x]:8050
        Member [x.x.x.x]:8050 this
}

[main] INFO com.hazelcast.core.LifecycleService - [x.x.x.x]:8050 [dev] [3.5.3] Address[x.x.x.x]:8050 is STARTED
[main] INFO org.apache.shiro.config.IniSecurityManagerFactory - Realms have been explicitly set on the SecurityManager instance - auto-setting of realms will not occur.
Open your web browser and navigate to https://127.0.0.1:8113/

Answer 1

I think my error was in my actual Java code, not in the configuration files:

In all my initial tests, running the service on a single server, I used this to get the current Shiro user:

Subject currentUser = SecurityUtils.getSubject();

However, the session key produced by this is the same for the same user on one server, but different for the same user on different servers; and the session key is used to identify a session in the cache. I was originally hoping to be able not having to pass a session key along with a REST call. but if I explicitly pass the session key and then use this to get the current user, then sessions can successfully be recognized in different servers:

Subject currentUser = null;
if (parameters.containsKey("session")) {
    sId = parameters.get("session");
    currentUser = new Subject.Builder().sessionId(sId).buildSubject();
} else {
    currentUser = SecurityUtils.getSubject();
}

(I am using a Map to store all URL parameters before reaching this code snippet)

Follow-up question: is is possible to share Shiro sessions without cookies or explicitly passing the session key in other ways from the client to the server?