Issue with rising heap size in a spring boot app (running in k8s)

Question

Description of the service:

This spring boot application is essentially a web socket proxy for charging stations (using OCPP protocol over websocket) running as multiple nodes behind a load balancer, to basically hold/keep open the connections, so that other microservices(actual consumers of the messages) which connect to this proxy over rabbitmq can be redeployed and maintained.

Heap dump:

These devices (tens of thousands) tend to connect and disconnect very often. The issue is that the

sessions (attribute) in org.apache.catalina.session.StandardManager

keeps adding sessions (more than 700 000) but not removing them (in a span of 7 days). see picture (StandartManager)

I also looked into

sessions (attribute) in org.apache.tomcat.websocket.server.WsServerContainer

which holds the websocket sessions (at the time of dump: 523) which was correct. see picture (WsServerContainer)

The application also keeps the active sessions (per node) in a concurrent map for own purposes:

sessions (attribute) in WebsocketServerService

at the time of dump: 530 (which correlates with the above sessions, do not mind the small difference, it's ok) see picture (WebsocketServerService)

So basically the expectation would be that all sessions attributes would hold almost the same amount of sessions, which is this case would be around 530 and NOT more than 700 000.

Question:

What might cause the everlasting rising of the standard sessions? I'm not familiar with inner workings of tomcat, if a websocket connection is "baked" by a standart session (at least at the beginning of the connection) and not properly closing the socket (by both parties) could cause that the standard sessions will stick around and are not collected after a while. These are just assumptions.

One version of the possible cause (WIP) is that we are using the standard WebSecurityConfigurerAdapter

@Configuration
@EnableWebSecurity
@AllArgsConstructor
public class SecurityConfig extends WebSecurityConfigurerAdapter {

    private final WebsocketServerProperties websocketServerProperties;
    private final StationAuthenticationProvider stationAuthenticationProvider;
    private final AuthenticationEntryPoint authEntryPoint;

    @Override
    public void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth.authenticationProvider(stationAuthenticationProvider);
    }

    @Override
    @SuppressWarnings("squid:S4502")
    protected void configure(HttpSecurity http) throws Exception {
        http.csrf().disable()
                .httpBasic()
                .authenticationEntryPoint(authEntryPoint)
                .and()
                .antMatcher(websocketServerProperties.getWebsocketPathVpn() + "/**").anonymous()
                .and()
                .antMatcher(websocketServerProperties.getWebsocketPathInternet() + "/**")
                .authorizeRequests()
                .anyRequest()
                .authenticated();
    }

}

and not the one mentioned here https://docs.spring.io/spring-security/reference/servlet/integrations/websocket.html#websocket-configuration

So if the station connects and there is an error for example 401, the socket is not properly closed, and the sessions increase without being properly closed and collected by GC (assumtion).

The implementations of the websocket controller is a standard one (at least by the reference doc):

@Configuration
@EnableWebSocket
@RequiredArgsConstructor
@EnableConfigurationProperties(WebsocketServerProperties.class)
public class WebsocketServerConfig implements WebSocketConfigurer {
    public static final String[] SUPPORTED_PROTOCOLS = new String[]{.....};

    private final WebsocketServerProperties websocketServerProperties;
    private final WebsocketHandShakeInterceptor websocketHandShakeInterceptor;
    private final StationSocketHandler stationSocketHandler;

    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(stationSocketHandler,
                        websocketServerProperties.getWebsocketPathVpn() + "/*",
                        websocketServerProperties.getWebsocketPathInternet() + "/*")
                .addInterceptors(websocketHandShakeInterceptor)
                .setHandshakeHandler(handshakeHandler())
                .setAllowedOrigins("*");
    }

    private HandshakeHandler handshakeHandler() {
        DefaultHandshakeHandler handler = new DefaultHandshakeHandler();
        handler.setSupportedProtocols(SUPPORTED_PROTOCOLS);
        return handler;
    }

    @Bean
    public ServletServerContainerFactoryBean createWebSocketContainer() {
        ServletServerContainerFactoryBean container = new ServletServerContainerFactoryBean();
        container.setMaxTextMessageBufferSize(websocketServerProperties.getMaxTextMessageBufferSize());
        container.setMaxSessionIdleTimeout(websocketServerProperties.getMaxSessionIdleTimeout());
        return container;
    }

}

@Slf4j
@Component
@RequiredArgsConstructor(onConstructor = @__({@Autowired}))
public class StationSocketHandler extends TextWebSocketHandler implements SubProtocolCapable {

    public static final String DEVICE_ID = "DEVICE_ID";
    public static final String CONNECTION_ROUTE = "CONNECTION_ROUTE";

    private final OcppMessageParser ocppMessageParser;
    private final WebsocketServerService websocketServerService;
    private final PingPongService pingPongService;

    @Override
    public void handleTextMessage(final WebSocketSession session, TextMessage textMessage) {
        OcppMessage message = ocppMessageParser.parsePayloadWithSessionId(session.getId(), textMessage.getPayload());
        websocketServerService.processMessageFromStation(message);
    }

    @Override
    public void afterConnectionEstablished(WebSocketSession session) {
        Map<String, Object> attributes = session.getAttributes();
        String deviceId = (String) attributes.get(DEVICE_ID);
        String connectionRoute = (String) attributes.get(CONNECTION_ROUTE);
        websocketServerService.connect(Route.toRoute(connectionRoute), deviceId, session);
    }

    @Override
    public void afterConnectionClosed(WebSocketSession session, @NonNull CloseStatus status) {
        log.info("WebSocketSession[{}][{}] closed with status {} and attributes {}",
                session.getId(), session.isOpen(), status, session.getAttributes());
        websocketServerService.disconnectSessionId(session.getId());
    }

    @Override
    protected void handlePongMessage(WebSocketSession session, PongMessage message) throws Exception {
        pingPongService.handlePong(session, message);
    }

    @Override
    public @NonNull
    List<String> getSubProtocols() {
        return Arrays.asList(WebsocketServerConfig.SUPPORTED_PROTOCOLS);
    }

}

Any suggestion, idea what could cause this would be appreciated.

I run some load tests to see if the sessions are rising and not correlating with the real state. The heap dumps after test were (at least for now) not very helpful. Also tried so simulate the errors, with no visible outcome.

Answer 1

I was able to find out what was the cause, so here the findings:

• if you use the default configuration of websocket handler in a spring boot application it's provided from tomcat, as the underlying provider
• we used the default @EnableWebSecurity => which means that for each websocket session there is also a classic StandardHttpSession created which is used for the authentication part
• the default timeout for the StandartHttpSession is 60s (server.servlet.session.timeout=60) In case of tomcat these are always multiples of 60 => 60,120,180.... But in the production (without my knowledge) somebody set this property to -1 => which means the session never expires (this solved the first visible problem of losing the web socket connections, but caused one new problem which was seen after a year(now) due to increased load)
• so the mechanics of the problem is, that after the timeout of the StandartHttpSession the "underlying" websocket session is closed EVEN-THOUGH were is visible communication.
• the issue is that the "activity" in the websocket session is not propagated to the above StandardHttpSession => that means the http session comes to timeout and closes the underlying websocket session

Solution for now (I need to test it) is to somehow overcome this 2 session constellation, or use a different provider like jetty, undertow.... or a hack to pass the reference of the httpsession to the websocket session as an attribute and on events onConnected set the timeout to -1 and after the session is closed set it to 60s => so that after a minute the session is gone.

The fix was to enforce the session policy to NEVER!

protected void configure(HttpSecurity http) throws Exception {
    http.csrf().disable()
    http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.NEVER)
        .and().csrf().disable()