Packet missing when both parties sending at the same time in a TCP connection

Question

I am coding with a tcp connection. But I observed some packet loss when both side sending/receiving messages. I can't figure out what's wrong here.

Below is a minimum example. I use gcc(with no flag/option) to compile. Each time in the for loop the two parties sends a 128 byte message to each other. Which contains a sequence num(the first byte). But it seems that some times the sent message are always missing.

Server:

#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <netdb.h>
#include <strings.h>
#include <stdlib.h>
void error(const char *msg)
{
    printf("%s\n",msg);
    exit(0);
}
FILE* server_open_socket(int portno)
{
    int sockfd, newsockfd;
    socklen_t clilen;
    struct sockaddr_in serv_addr, cli_addr;
    sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd < 0)
        error("ERROR opening socket");
    bzero((char *) &serv_addr, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = INADDR_ANY;
    serv_addr.sin_port = htons(portno);
    int on=1;
    if((setsockopt(sockfd,SOL_SOCKET,SO_REUSEADDR,&on,sizeof(on)))<0)
    {
        perror("setsockopt failed");
        exit(EXIT_FAILURE);
    }
    if (bind(sockfd, (struct sockaddr *) &serv_addr,
             sizeof(serv_addr)) < 0)
        error("ERROR on binding");
    listen(sockfd,5);
    clilen = sizeof(cli_addr);
    newsockfd = accept(sockfd,
                       (struct sockaddr *) &cli_addr,
                       &clilen);
    if (newsockfd < 0)
        error("ERROR on accept");
    close(sockfd);
    FILE *stream;
    stream = fdopen(newsockfd, "wb+");
    printf("connected Server at %d\n",portno);
    return stream;
}
int main()
{
    FILE *S;
    S=server_open_socket(12345);
    char sbuffer[128],rbuffer[128];
    for(int i=0;i<128;i++)
        sbuffer[i]=0;
    for(int i=0;i<256;i++)
    {
        rbuffer[0]=0;
        sbuffer[0]=1+i%255;
        int a=fwrite(sbuffer,sizeof(unsigned char),128,S);
        int b=fread(rbuffer,sizeof(unsigned char),128,S);
        if((a!=128)||(b!=128))
            printf("Network error!\n");
        if(rbuffer[0]!=sbuffer[0])
        {
            printf("msgerror! %d %d\n",rbuffer[0],sbuffer[0]);
            exit(0);
        }
   }

}

client:

#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <netdb.h>
#include <strings.h>
#include <stdlib.h>
void error(const char *msg)
{
    printf("%s\n",msg);
    exit(0);
}
FILE* client_open_socket(int portno)
{
    int sockfd;
    struct sockaddr_in serv_addr;
    struct hostent *server;
    sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd < 0)
        error("ERROR opening socket");
    server = gethostbyname("localhost");//gethostbyname(argv[1]);
    bzero((char *) &serv_addr, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    bcopy((char *)server->h_addr,
          (char *)&serv_addr.sin_addr.s_addr,
          server->h_length);
    serv_addr.sin_port = htons(portno);
    while(connect(sockfd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)) == -1) {
        sleep(1000);
    }
    FILE *stream;
    stream = fdopen(sockfd, "wb+");
    printf("connected Clinet at %d\n",portno);
    return stream;
}
int main()
{
    FILE *S;
    S=client_open_socket(12345);
    char sbuffer[128],rbuffer[128];
    for(int i=0;i<128;i++)
        sbuffer[i]=0;
    for(int i=0;i<256;i++)
    {
        rbuffer[0]=0;
        sbuffer[0]=1+i%255;
        int a=fwrite(sbuffer,sizeof(unsigned char),128,S);
        int b=fread(rbuffer,sizeof(unsigned char),128,S);
        if((a!=128)||(b!=128))
            printf("Network error!\n");
        if(rbuffer[0]!=sbuffer[0])
        {
            printf("msgerror! %d %d\n",rbuffer[0],sbuffer[0]);
            exit(0);
        }
    }
}

Here is a possible output:

server:

Network error!
msgerror! 0 3

client:

msgerror! 3 2

It seems that the message starting with "2" from the server is missing.

This issue seems to disappear if I let the two parties send message in order(server sends and then listens, client listens then sends). But I don't know why either.

When it comes to multiple parties, for example, three machines/hosts/parties, each of them has a distinct message for each other party: 6 messages in all. How should I handle such task?

Answer 1

You're totally confusing the stdio library.

You can't use a buffered stream in both directions on the same socket at the same time. The buffered stream mechanism is designed with a certain model in mind: that of an ordinary file on disk. The socket is not like that. That is, the library is designed for a model where the "write" side and the "read" side share a single underlying file pointer and you can both read and write into the same area of the data.

Think about how this would work if you were operating on an ordinary file on disk and contrast that with the socket. You fread 128 bytes from the "file". This causes the library to actually read (as in the read(2) system call) at least 128 bytes -- but it will try to read more than that normally, up to some large buffer size -- into its internal buffer. It will then transfer 128 bytes into your buffer, and set its internal file offset to 128. If you then write 128 bytes, the stdio library will, I believe, overwrite any bytes in its internal buffer starting at offset 128 for a length of 128.

If it had read more than 128 bytes initially, it would now need to reposition the underlying file pointer before it could actually flush that data "to disk" (ie call write(2) ). I'm not sure exactly how much it would write then, but it will attempt to keep a coherent view of that data in its internal buffer -- but again it's assuming a model which is totally different than a bidirectional socket data stream. So, prior to doing the write, it will attempt to reposition the file pointer in preparation for updating the file and because the "file" is actually a socket, the lseek(2) system call fails (which explains the "network error" you're seeing).

You can create two separate buffered streams on the socket, one for reading ( fdopen(sockfd, "rb") ) and one for writing ( fdopen(sockfd, "wb") ). I don't know if that is guaranteed to work by the library standardization -- probably not -- but since each buffered stream is totally independent and since you're only reading from one and only writing to the other, there aren't any issues with the library needing to adjust its file offset in a way that affects both directions, and the library can treat each "stream" pretty much exactly as a file on disk that is slowly growing.

If you try that, you'll need to add some fflush(3) calls in order to force the library to actually send the buffered data to the peer. Because otherwise, it doesn't know it needs to do that.