Why does it take so long to send TCP packets to localhost?

Question

I wrote a simple C++ client/server pair. The server just forks a process on socket accept and then waits for a packet from the client and then responds with another packet. The client just sends a packet to server and then waits for a reply. I have timing code in the client right before the send and then after the receive.

I am running both the server and client on my local box and connecting the client to local host.

In my timings the median latency seems to be around 2 milliseconds. Given that I am not really sending anything out on the network. The 2 millisecond latency seems awfully high to me.

Can anyone explain why I see such a high latency or if this amount of time is realistic for the loopback address?

I am on Linux Ubuntu 12.04. I am directly using the TCP socket system calls rather than any wrapper (ie. accept, listen, send, receive).

Server body:

while (1) 
{   
    ++msgNum;

    sin_size = sizeof their_addr; 
    new_fd = accept(sockfd, (struct sockaddr*) &their_addr, &sin_size);
    if (new_fd == -1) 
    {   
        perror("accept");
        continue; 
    }   

    inet_ntop(their_addr.ss_family, get_in_addr((struct sockaddr*) &their_addr), 
        s, sizeof s); 
    printf("server: got connection from %s\n", s); 

    if (!fork())
    {   
        close(sockfd); // child doesn't need the listener

        MyMsg msg;
        strcpy(msg.buf, "Hello world");

        for (int i = 1; i <= NUM_TEST_MSGS; ++i)
        {   
            msg.id = i;

            int numbytes = 0;
            int bytesRecv = 0;

            while (numbytes < MSG_LEN)
            {   
                int sendSize = MSG_LEN - numbytes;
                if ((bytesRecv = send(new_fd, ((char*) &msg) + numbytes, 
                        sendSize, 0)) == -1) 
                {   
                    perror("send");
                    exit(1);
                }   
                numbytes += bytesRecv;
            }   

            assert(numbytes == MSG_LEN);

            //printf("Server sent %d num bytes\n", numbytes);
        }   

        printf("Server finished sending msgs.\n");

        close(new_fd);
        exit(0);
    }   
    close(new_fd);
} 

Client body:

for (int i = 1; i <= NUM_TEST_MSGS; ++i)
{
    MyMsg msg;

    int numbytes = 0;
    int bytesRecv = 0;

    int start = rdTsc.Rdtsc();

    while (numbytes < MSG_LEN)
    {
        int recvSize = MSG_LEN - numbytes;
        if ((bytesRecv = recv(sockfd, ((char*) &msg) + numbytes, recvSize, 0)) == -1)
        {
            perror("recv");
            exit(1);
        }

        numbytes += bytesRecv;
    }

    int end = rdTsc.Rdtsc();

    perfCounter.Track(end - start);

    if (numbytes != MSG_LEN)
    {
        printf("COMP FAILED: %d %d\n", numbytes, MSG_LEN);
    }

    assert(numbytes == MSG_LEN);

    if (i != msg.id)
    {
        printf("Msg %d id %d \n", i, msg.id);
    }

    //if (numbytes != MSG_LEN) printf("GOT WEIRD SIZE %d\n", numbytes);
    assert(msg.id == i);

    //printf("client: received %d num bytes id %d body '%s'\n", numbytes, msg.id, msg.buf);

    if (i % 1000 == 0)
    {
        printf("Client: Received %d num msgs.\n", i);
    }
}

printf("Client: Finished successfully.\n");

close(sockfd);

Answer 1

2ms certainly sounds high for something that probably never even left the kernel buffers. I suspect it might actually be inaccuracies in the function used for time-stamping.