C socket : Non blocking way to read \n separated commands

Question

As a school project, I have to recode a IRC server but I'm stuck on a problem. What I am trying to do is to receive and execute client's commands without blocking (as I have many clients to serve).

Edit: The use of non blocking socket and fork() is forbidden for this project

About the commands:

1. They are "\\r\\n" separated
2. They are 512 char max

My first attempt was to loop with a getline. It worked perfectly but only for one client (as the getline block when their is noting more to read instead of passing to the next client)

bool     recv_cmd(t_hdl *hdl)                                               
{                                                                                             
  char          *raw;                                                                         
  size_t        len;                                                                          
  FILE          *input_stream;                                                                
  ssize_t       nread;                                                                        

  len = 0;                                                                                    
  raw = NULL;                                                                                 
  if ((input_stream = fdopen(dup(hdl->sender->fd), "r")) == NULL)                             
    return (false);                                                                           
  while ((nread = getline(&raw, &len, input_stream)) > 0)                                     
    {                                                                                         
      printf("%lu\n", nread);                                                                 
      parse_cmd(hdl, raw);                                                                    
      exec_cmd(hdl);                                                                                                                                                 
    }                                                                                         
  fclose(input_stream);                                                                       
  return (true);                                                                              
} 

If I remove the getline from the loop like so, it work for all clients but only the first command from the client is executed (for example, if the client send "command1\\r\\ncommand2\\r\\n", only command1 is executed)

bool     recv_cmd(t_hdl *hdl)                                               
{                                                                                             
  char          *raw;                                                                         
  size_t        len;                                                                          
  FILE          *input_stream;                                                                

  len = 0;                                                                                    
  raw = NULL;                                                                                 
  if ((input_stream = fdopen(dup(hdl->sender->fd), "r")) == NULL)                             
    return (false);                                                                           
  if (getline(&raw, &len, input_stream) != -1)                                                
    {                                                                                         
      parse_cmd(hdl, raw);                                                                    
      exec_cmd(hdl);                                                                          
      //free(raw                                                                              
    }                                                                                         
  fclose(input_stream);                                                                       
  return (true);                                                                              
}          

I also tried to remove the fclose() so that when we read command1, command2 stay in the stream buffer but it didn't work either.

The subject of the project also say that " use circular buffers in order to secure and optimize the various commands and responses that are being sent and received. ".

How should I do it ? And what are the advantages of using circular buffer over my getline in this case ?

Answer 1

Since you used getline() , I'm assuming you are relying on POSIX.1 features; in this case, I'd recommend using a dedicated thread for receiving messages from all connected clients.

Rather than just read additional data from a per-client dynamic buffer, I would put incoming messages into a chain:

#define MAX_INCOMING_LEN  512

struct incoming_message {
    struct incoming_message  *next;
    size_t                    len;
    char                      data[MAX_INCOMING_LEN];
}

The client structure needs a temporary buffer of at least MAX_INCOMING_LEN chars (since there are no guarantees that a recv() or read() from a stream socket provides a complete message, or just a single message). If a separate thread is reading the messages, then you also need locking to protect the message chain from concurrent access:

struct client {
    int                      socketfd;
    char                     received[MAX_INCOMING_LEN];
    size_t                   received_len;

    pthread_mutex_t          incoming_lock;
    struct incoming_message *incoming_next;
    struct incoming_message *incoming_last;
};

The function that receives new messages appends them to the list thus, in pseudocode:

Construct and fill in struct incoming_message *msg
Lock incoming_lock mutex
Set msg->next = NULL
If incoming_last != NULL:
    Set incoming_last->next = msg
    Set incoming_last = msg
Else
    Set incoming_next = msg
    Set incoming_last = msg
End If
Unlock incoming_lock mutex

Using two pointers, incoming_next and incoming_last , means we don't need to scan through the list when appending to it. A function to grab the next incoming message, given a client c , in pseudocode is something like

Function next_message(struct client *c)
{
    Lock c->incoming_lock mutex
    If c->incoming_next != NULL:
        struct incoming_message *msg = c->incoming_next;
        If msg->next != NULL
            Set incoming_next = msg->next
            Set msg->next = NULL
        Else:
            Set incoming_next = NULL
            Set incoming_last = NULL
        End If
        Unlock c->incoming_lock mutex
        Return msg
    Else:
        Unlock c->incoming_lock mutex
        Return NULL
    End If
}

Note that for outgoing messages, I'd use a completely different structure, because you usually send the exact same message to a number of clients. There are at least two completely different approaches to this, but the OP did not ask about these, so I shall omit my ruminations about those.

The incoming data worker, or socket reader thread, is the only one that touches the per-client received[] buffer, so it does not need any locking.

Let's assume you have the following global variables:

static pthread_mutex_t   received_lock = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t    received_more = PTHREAD_COND_INITIALIZER;
static long              received_gen  = 0L;

In pseudocode, the socket reader thread does the following work in a loop:

Use select() or poll() to find out which clients' sockets have unread data
Lock received_lock mutex
Set have_received = 0
For each client whose socket has unread data:
    Try receiving as much as is free in received[] buffer
    If new data received:
        Increment received_len by the received amount
        Increment have_received by 1
        If a separator exists in received[0..received_len-1]:
            Let N be the offset of the character following the separator
            Grab or allocate a new incoming_message structure
            Copy the first N chars of received[] to the new structure
            Lock the incoming_lock mutex
            Prepend the structure to the singly-linked list
            Unlock the incoming_lock mutex
            If N < received_len:
                memmove(received, received + N, received_len - N)
                received_len -= N
            Else:
                received_len = 0
            End If
        End If
    End If
End If
If have_received > 0:
    Increment received_gen by 1
    Signal on received_more condition variable
End If
Unlock received_lock mutex

The purpose of the received_lock , received_wait , and received_gen is to avoid busy-looping when no new messages come in.

Assuming you use your main thread to handle each incoming message, it will have a loop, with the body of the loop something like this:

Lock received_lock mutex
before_gen = received_gen
Unlock received_lock mutex

Set msg_count = 0
For each client:
    Lock client->incoming_lock
    If the list is not empty:
        Increment msg_count by 1
        Grab the last message in the list
        Unlock client->incoming_lock

        Process the message

    Else:
        Unlock client->incoming_lock
    End If
End For

If msg_count == 0:
    Lock received_lock mutex
    after_gen = received_gen
    If after_gen == before_gen:
        pthread_cond_wait(received_more, received_lock)
    End if
    Unlock received_lock mutex
End If

We do not want to hold received_lock for any length of time, because that blocks receiving of new messages. Instead, we use received_gen as a generation counter: If there was no work to do, we check if the generation counter has changed. If it has, there may be more work to do, so we continue to the next iteration of the main loop. Otherwise, and note that we still hold the mutex, we wait for a signal on the condition variable.