Scaling gRPC bidirectional streaming chat service

Question

I'm drafting a chat service in gRPC java with bidirectional streaming. Simplified flow is like below,

1. When user joins chat service, user's StreamObserver will be stored in a chat room repository ie a simple HashMap holding userId - StreamObserver in the server.
2. After a while, when user sends a chat message, server receives the request and broadcasts the message to all the users in the chat room by iterating StreamObserver s stored in the chat room repository and calling onNext method.

This works fine when there's only 1 server existing, however once scaled out to multiple servers, clients' StreamObserver s will be stored in a specific server and will not exist in other servers as gRPC opens a single HTTP connection to the server initially connected.

What I need is sending the message to all the users in the same chat room by getting all StreamObserver s scattered around the servers, does anyone have good experience with this kind of situation? I tried to store StreamObserver in a single storage however as it isn't serializable, I couldn't store it in a shared storage like redis.

Answer 1

I implemented a chat using gRPC and 3 servers with load balance. The first thing to achieve the load balance is to use a ManagedChannel with defaultLoadBalancingPolicy . In my case I used round_robin policy. And create the channel using a MultiAddressNameResolverFactory with the host and ports of the three servers. Here I create a client chat for Alice. Then You copy this class and create a client chat for Bob. This should already do the load balance that you asked.

public class ChatClientAlice {
    private NameResolver.Factory nameResolverFactory;
    private ManagedChannel channel;

    public static void main(String[] args) {
        ChatClientAlice chatClientAlice = new ChatClientAlice();
        chatClientAlice.createChannel();
        chatClientAlice.runBiDiStreamChat();
        chatClientAlice.closeChannel();
    }

    private void createChannel() {
        nameResolverFactory = new MultiAddressNameResolverFactory(
                new InetSocketAddress("localhost", 50000),
                new InetSocketAddress("localhost", 50001),
                new InetSocketAddress("localhost", 50002)
        );
        channel = ManagedChannelBuilder.forTarget("service")
                .nameResolverFactory(nameResolverFactory)
                .defaultLoadBalancingPolicy("round_robin")
                .usePlaintext()
                .build();
    }

    private void closeChannel() { channel.shutdown(); }

    private void runBiDiStreamChat() {
        System.out.println("creating Bidirectional stream stub for Alice");
        EchoServiceGrpc.EchoServiceStub asyncClient = EchoServiceGrpc.newStub(channel);
        CountDownLatch latch = new CountDownLatch(1);

        StreamObserver<EchoRequest> requestObserver = asyncClient.echoBiDi(new StreamObserver<EchoResponse>() {
            @Override
            public void onNext(EchoResponse value) { System.out.println("chat: " + value.getMessage()); }

            @Override
            public void onError(Throwable t) { latch.countDown(); }

            @Override
            public void onCompleted() { latch.countDown(); }
        });

        Stream.iterate(0, i -> i + 1)
                .limit(10)
                .forEach(integer -> {
                    String msg = "Hello, I am " + ChatClientAlice.class.getSimpleName() + "! I am sending stream message number " + integer + ".";
                    System.out.println("Alice says: " + msg);
                    EchoRequest request = EchoRequest.newBuilder()
                            .setMessage(msg)
                            .build();
                    requestObserver.onNext(request);
                    // throttle the stream
                    try { Thread.sleep(5000); } catch (InterruptedException e) { }
                });
        requestObserver.onCompleted();
        System.out.println("Alice BiDi stream is done.");
        try {
            // wait for the time set on the stream + the throttle
            latch.await((5000 * 20), TimeUnit.MILLISECONDS);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

On the server service you will have to use a singleton to store the StreamObserver s every time that you receive a new request from new clients. Instead of returning the message to a single observer responseObserver.onNext(response); you will iterate all observers and send the message to all singletonObservers.getObservers().forEach(.... . Although this has nothing to do with the load balance strategy I thought that it is worthwhile to post because if you don't implement it well your clients will not receive messages from other clients.

public class ChatServiceImpl extends EchoServiceGrpc.EchoServiceImplBase {

    private final String name;
    private final SingletlonChatStreamObserver singletonObservers;

    ChatServiceImpl(String name) {
        this.name = name;
        this.singletonObservers = SingletlonChatStreamObserver.getInstance();
    }

    @Override
    public StreamObserver<EchoRequest> echoBiDi(StreamObserver<EchoResponse> responseObserver) {
        System.out.println("received bidirectional call");

        singletonObservers.addObserver(responseObserver);
        System.out.println("added responseObserver to the pool of observers: " + singletonObservers.getObservers().size());

        StreamObserver<EchoRequest> requestObserver = new StreamObserver<EchoRequest>() {
            @Override
            public void onNext(EchoRequest value) {
                String msg = value.getMessage();
                System.out.println("received message: " + msg);
                EchoResponse response = EchoResponse.newBuilder()
                        .setMessage(msg)
                        .build();
                // do not send messages to a single observer but to all observers on the pool
                // responseObserver.onNext(response);
                // observers.foreach...
                singletonObservers.getObservers().forEach(observer -> {
                    observer.onNext(response);
                });
            }

            @Override
            public void onError(Throwable t) {
                // observers.remove(responseObserver);
                singletonObservers.getObservers().remove(responseObserver);
                System.out.println("removed responseObserver to the pool of observers");
            }

            @Override
            public void onCompleted() {
                // do not complete messages to a single observer but to all observers on the pool
                // responseObserver.onCompleted();
                // observers.foreach
                singletonObservers.getObservers().forEach(observer -> {
                    observer.onCompleted();
                });

                // observers.remove(responseObserver);
                System.out.println("removed responseObserver to the pool of observers");
            }
        };
        return requestObserver;
    }
}

and this is my SingletlonChatStreamObserver to have only one of this object for all 3 servers:

public class SingletlonChatStreamObserver implements Serializable {

    private static volatile SingletlonChatStreamObserver singletonSoleInstance;
    private static volatile ArrayList<StreamObserver<EchoResponse>> observers;

    private SingletlonChatStreamObserver() {
        //Prevent form the reflection api.
        if (singletonSoleInstance != null) {
            throw new RuntimeException("Use getInstance() method to get the single instance of this class.");
        }
    }

    public static SingletlonChatStreamObserver getInstance() {
        if (singletonSoleInstance == null) { //if there is no instance available... create new one
            synchronized (SingletlonChatStreamObserver.class) {
                if (singletonSoleInstance == null) {
                    observers = new ArrayList<StreamObserver<EchoResponse>>();
                    singletonSoleInstance = new SingletlonChatStreamObserver();
                }
            }
        }
        return singletonSoleInstance;
    }

    //Make singleton from serializing and deserialize operation.
    protected SingletlonChatStreamObserver readResolve() {
        return getInstance();
    }

    public void addObserver(StreamObserver<EchoResponse> streamObserver) {
        observers.add(streamObserver);
    }

    public ArrayList<StreamObserver<EchoResponse>> getObservers() {
        return observers;
    }
}

I will commit the complete code on my explore-grpc project.