How to shutdown gRPC server from Client (using RPC function)

Question

I'm using gRPC for inter-process communication between C++ App (gRPC Server) and Java App (gRPC Client). Everything run on one machine. I want to provide client possibility to shut down the server. My idea is to add RPC function to service in proto which would do it.

The C++ Implementation would be:

class Service : public grpcGeneratedService
{
public:
......
private:  
    grpc::Server* m_pServer;
};
grpc::Status Service::ShutDown(grpc::ServerContext* pContext, const ShutDownRequest* pRequest, ShutDownResponse* pResponse)
{
    if (m_pServer)
        m_pServer->Shutdown();
    return grpc::Status(grpc::StatusCode::OK, "");
}

However the ShutDown blocks until all RPC calls are processed what means dead-lock. Is there any elegant way how to implement it?

Answer 1

I'm using a std::promise with a method almost exactly like yours.

// Somewhere in the global scope :/
std::promise<void> exit_requested;

// My method looks nearly identical to yours 
Status CoreServiceImpl::shutdown(ServerContext *context, const SystemRequest *request, Empty*)
{
  LOG(INFO) << context->peer() << " - Shutdown request acknowledged.";
  exit_requested.set_value();
  return Status::OK;
}

In order to make this work, I call server->Wait() in a second thread and wait on the future for the exit_requested promise to block a shutdown call:

auto serveFn = [&]() {
  server->Wait();
};

std::thread serving_thread(serveFn);

auto f = exit_requested.get_future();
f.wait();
server->Shutdown();
serving_thread.join();

Once I had this I was also able to support a clean shutdown via signal handlers as well:

auto handler = [](int s) {
  exit_requested.set_value();
};
std::signal(SIGINT, handler);
std::signal(SIGTERM, handler);
std::signal(SIGQUIT, handler);

I've been satisfied with this approach so far and it's kept me within the bounds of gRPC and the standard c++ libs. Rather than use some globally scoped promise (I have to declare it as an external in my service implementation source) I should probably think of something more elegant.

One thing to note here is that setting the value of the promise more than once will throw an exception. This could happen if you somehow send the shutdown message and also pkill -2 my_awesome_service at the same time. I actually ran into this when there was a deadlock in my persistence layer preventing shutdown from finishing, when I tried to send a SIGINT again the service aborted instead! For my needs this is still an acceptable solution but I'd love to hear about alternatives that work around or solve that little problem.

Answer 2

You can create an std::function from the ShutDown() handler and run that function in a separate thread (or threadpool). This will allow decoupling the handling of the RPC from the execution of the shutdown logic and eliminate the deadlock.