POCO connection gets stuck

Question

I am using POCO reactor pattern for handling incoming tcp connections. Connections might take from couple of seconds to minutes depending on the request type as follows:

  try{
    ServerSocket serverSocket(port);
    reactor = new SocketReactor();
    ParallelSocketAcceptor<BFSTcpServiceHandler,SocketReactor> acceptor(serverSocket, *reactor);
    //Start Reactor
    reactor->run();
  }catch(Exception&e){
    LOG(ERROR)<<"ERROR in initializing TCPServer:"<<e.message();
    return;
  }

And here is the Handler:

BFSTcpServiceHandler::BFSTcpServiceHandler(StreamSocket& _socket,
    SocketReactor& _reactor): socket(_socket),reactor(_reactor) {
  //Set Keeep Alive for socket
  socket.setKeepAlive(false);

  //Register Callbacks
  reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    ReadableNotification>(*this, &BFSTcpServiceHandler::onReadable));
  /*reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    WritableNotification>(*this, &BFSTcpServiceHandler::onWriteable));*/
  reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    ShutdownNotification>(*this, &BFSTcpServiceHandler::onShutdown));
  reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    ErrorNotification>(*this, &BFSTcpServiceHandler::onError));
  reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    TimeoutNotification>(*this, &BFSTcpServiceHandler::onTimeout));
  /*reactor.addEventHandler(socket, NObserver<BFSTcpServiceHandler,
    IdleNotification>(*this, &BFSTcpServiceHandler::onIdle));*/
}

BFSTcpServiceHandler::~BFSTcpServiceHandler() {
  //Unregister Callbacks
  reactor.removeEventHandler(socket, NObserver<BFSTcpServiceHandler,
    ReadableNotification>(*this, &BFSTcpServiceHandler::onReadable));
    ...
  //Close socket
  try {
    socket.close();
  }catch(...){}
}

void BFSTcpServiceHandler::onReadable(
    const Poco::AutoPtr<Poco::Net::ReadableNotification>& pNf) {
  //LOG(ERROR)<<"onReadable:"<<socket.peerAddress().toString();
  try{
   //Read and process request
  } catch(Exception &e){
    LOG(ERROR)<<"Error in reading request:"<<e.message();
    delete this;
  }
  //So after a connection is served just close it!
  delete this;
}

void BFSTcpServiceHandler::onShutdown(
    const Poco::AutoPtr<Poco::Net::ShutdownNotification>& pNf) {
  LOG(ERROR)<<"onShutdown:"<<socket.peerAddress().toString();

  //Call destructor of this class
  delete this;
}

void BFSTcpServiceHandler::onWriteable(
    const Poco::AutoPtr<Poco::Net::WritableNotification>& pNf) {
  static bool once = true;
  if(once) {
    LOG(ERROR)<<"onWritable:"<<socket.peerAddress().toString()<<" keepAlive?"<<socket.getKeepAlive()<<" isBlocking?"<<socket.getBlocking()<<" noDeley?"<<socket.getNoDelay();
    once = false;
  }
}

void BFSTcpServiceHandler::onTimeout(
    const Poco::AutoPtr<Poco::Net::TimeoutNotification>& pNf) {
  LOG(ERROR)<<"\nTIMEOUT! onTimeout:"<<socket.peerAddress().toString();
}

void BFSTcpServiceHandler::onError(
    const Poco::AutoPtr<Poco::Net::ErrorNotification>& pNf) {
  LOG(ERROR)<<"\nERROR! onError:"<<socket.peerAddress().toString();
}

void BFSTcpServiceHandler::onIdle(
    const Poco::AutoPtr<Poco::Net::IdleNotification>& pNf) {
  LOG(ERROR)<<"\nIDLE! onIdle:"<<socket.peerAddress().toString();
}

The code works fine; however, after a while it gets stuck meaning that the server does accepts connections but onReadable is not called at all anymore. For example, after it gets stuck I can telnet to the server but when I send data onReadable is not fired. Using netstat I realized some data are being kept in the RCV_QEUEUE and reactor does not fire onReadable event.

I thought it's due to hitting connection/file limits of systems but it is not actually many connections open when the system gets stuck.

Any comment or help is appreciated.

Thanks,

Answer 1

The problem was using a faulty NIC/driver. I changed the code to regular POSIX sockets and had the same issue and switching the NIC solved the issue. I am not sure if it was a driver or hardware issue.