Google Cloud Pub Sub memory leak on re-deploy (Netty based)

Question

My tomcat web service uses realtime developer notifications for Android , which requires Google Cloud Pub Sub. It is working flawlessly, all the notifications are received immediately. The only problem is that it uses too much RAM that causes the machine to respond very slowly than it is supposed to, and is not releasing it after undeploying the application. It uses HttpServlet (specifically Jersey which provides contextInitialized and contextDestroyed methods to set and clear references) and commenting the pub-sub code actually decreases a lot of the memory usage.

Here is the code for subscribing-unsubscribing for Android subscription notifications.

package com.example.webservice;

import com.example.webservice.Log;
import com.google.api.core.ApiService;
import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.cloud.pubsub.v1.Subscriber;
import com.google.common.collect.Lists;
import com.google.pubsub.v1.ProjectSubscriptionName;

import java.io.FileInputStream;

public class SubscriptionTest
{
    // for hiding purposes
    private static final String projectId1 = "api-000000000000000-000000";
    private static final String subscriptionId1 = "realtime_notifications_subscription";
    private static final String TAG = "SubscriptionTest";

    private ApiService subscriberService;
    private MessageReceiver receiver;

    // Called when "contextInitialized" is called.
    public void initializeSubscription()
    {
        Log.w(TAG, "Initializing subscriptions...");
        try
        {
            GoogleCredentials credentials1 = GoogleCredentials.fromStream(new FileInputStream("googlekeys/apikey.json"))
                    .createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
            ProjectSubscriptionName subscriptionName1 = ProjectSubscriptionName.of(projectId1, subscriptionId1);

            // Instantiate an asynchronous message receiver
            receiver =
                    (message, consumer) ->
                    {
                        consumer.ack();

                        // do processing
                    };

            // Create a subscriber for "my-subscription-id" bound to the message receiver
            Subscriber subscriber1 = Subscriber.newBuilder(subscriptionName1, receiver)
                    .setCredentialsProvider(FixedCredentialsProvider.create(credentials1))
                    .build();

            subscriberService = subscriber1.startAsync();
        }
        catch (Throwable e)
        {
            Log.e(TAG, "Exception while initializing async message receiver.", e);
            return;
        }
        Log.w(TAG, "Subscription initialized. Messages should come now.");
    }

    // Called when "contextDestroyed" is called.
    public void removeSubscription()
    {
        if (subscriberService != null)
        {
            subscriberService.stopAsync();
            Log.i(TAG, "Awaiting subscriber termination...");
            subscriberService.awaitTerminated();
            Log.i(TAG, "Subscriber termination done.");
        }

        subscriberService = null;
        receiver = null;
    }
}

And this is the statement after the application is undeployed. (Names may not match but it is not important)

org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application 
[example] created a ThreadLocal with key of type [java.lang.ThreadLocal] 
(value [java.lang.ThreadLocal@2cb2fc20]) and a value of type 
[io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap] 
(value [io.grpc.netty.shaded.io.netty.util.internal.InternalThreadLocalMap@4f4c4b1a]) 
but failed to remove it when the web application was stopped. 
Threads are going to be renewed over time to try and avoid a probable memory leak.

From what I've observed, Netty is creating a static ThreadLocal with a strong reference to the value InternalThreadLocalMap which seems to be causing this message to appear. I've tried to delete it by using some sort of code like this (probably it's overkill but none of the answers worked for me so far, and this isn't seem to be working either)

    InternalThreadLocalMap.destroy();
    FastThreadLocal.destroy();
    for (Thread thread : Thread.getAllStackTraces().keySet())
    {
        if (thread instanceof FastThreadLocalThread)
        {
            // Handle the memory leak that netty causes.
            InternalThreadLocalMap map = ((FastThreadLocalThread) thread).threadLocalMap();
            if (map == null)
                continue;

            for (int i = 0; i < map.size(); i++)
                map.setIndexedVariable(i, null);
            ((FastThreadLocalThread) thread).setThreadLocalMap(null);
        }
    }

After the undeploy (or stop-start) tomcat detects a memory leak if I click Find leaks (obviously). The problem is, the RAM and CPU that has been used is not released because apparently the subscription is not closed properly. Re-deploying the app causes the used RAM to increase even further on every action like, if it uses 200 MB ram at first, after 2nd deploy it increases to 400, 600, 800 which goes unlimited until the machine slows down enough to die.

It is a serious issue and I have no idea how to solve it, the stop methods are called as defined, awaitTerminated is also called which immediately executes (means that the interface is actually stopped listening) but it does not release the RAM behind it.

So far I've only seen questions about python clients ( ref 1 , ref 2 ) but nobody seems to be mentioning the Java client, and I'm kind of losing hope about using this structure.

I've opened an issue about this problem as well.

What should I do to resolve this issue? Any help is appreciated, thank you very much.

Answer 1

I don't know if it will fully fix your issue, but you appear to be leaking some memory by not closing the FileInputStream.

The first option is to extract the FileInputStream into a variable and call the close() method on it after you are done reading the content.

A second (and better) option to work with these kind of streams is to use try-with-resources. Since FileInputStream implements the AutoCloseable interface, it will be closed automatically when exiting the try-with-resources.

Example:

try (FileInputStream stream = new FileInputStream("googlekeys/apikey.json")) {
    GoogleCredentials credentials1 = GoogleCredentials.fromStream(stream)
            .createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));
    // ...
} catch (Exception e) {
    Log.e(TAG, "Exception while initializing async message receiver.", e);
    return;
}