Unix domain socket and write buffer overflow

Question

I use Unix domain sockets in one of my application and I am observing a very strange behavior while writing to a socket. A short description: there are 2 application - client and server where each one read and write at the same simultaneously, ie the socket works in full duplex mode. To prevent some overflow I check the write buffer:

ioctl(socket_handler, TIOCOUTQ, &pending)

But sometimes the result I get is very strange - 768, 1536, 2304 etc. On the one hand, this number does not look like a bytes count I write, on the other hand it looks a little suspicious - 0x300, 0x600 etc, in other words it looks like a flag or error code. The documentation says nothing about this.

To be consistent and prevent questions I provide a full code example to reproduce the issue:

#include <iostream>
#include <string>
#include <sys/socket.h>
#include <sys/un.h>
#include <linux/sockios.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <pthread.h>
#include <poll.h>

static int sock;
static socklen_t len;
static struct sockaddr_un addr;
static std::string path = "@testsock";
static pthread_t thread;
static bool running;
static pollfd polls[2] = {};
static char buf[200];
static int err;

bool InitSocket()
{
    sock = socket(AF_UNIX, SOCK_STREAM,0);
    memset(&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    strncpy(addr.sun_path, path.c_str(), path.size());
    *addr.sun_path = '\0';
    len = static_cast<socklen_t>(__builtin_offsetof(struct sockaddr_un, sun_path) + path.length());
    int opt_value = 1;
    err = setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, static_cast<void *>(&opt_value), sizeof(int));
    int on = 1;
    err = ioctl(sock, FIONBIO, &on);
    return true;
}

bool InitServer()
{
    polls[0].fd = sock;
    polls[0].events = POLLIN;

    err = bind(sock, reinterpret_cast<struct sockaddr *>(&addr), len);
    err = listen(sock, 10);
    return true;
}

bool InitClient()
{
    polls[1].fd = sock;
    polls[1].events = POLLIN;

    err = connect(sock, reinterpret_cast<struct sockaddr *>(&addr), len);
    return true;
}

uint32_t GetRand(uint32_t min, uint32_t max)
{
    return static_cast<uint32_t>(rand()) % (max - min) + min;
}
void *ReadThread(void*)
{
    while(running)
    {
        int status = poll(polls, 2, 1000);
        if(status > 0)
        {
            for(int i = 0;i < 2;i ++)
            {
                if(polls[i].revents == 0)
                    continue;

                if(i == 0)
                {
                    struct sockaddr_un cli_addr;
                    socklen_t clilen = sizeof(cli_addr);
                    int new_socket = accept(sock, reinterpret_cast<struct sockaddr *>(&cli_addr), &clilen);
                    int on = 1;
                    ioctl(new_socket, FIONBIO, &on);
                    polls[1].fd = new_socket;
                    polls[1].events = POLLIN;
                    std::cout << "client connected" << std::endl;
                }
                else
                {
                    ssize_t readBytes = read(polls[1].fd, &buf, 100);
                    if(readBytes > 0)
                    {
                        //std::string str(buf, readBytes);
                        std::cout << "         read " << readBytes << " bytes: " << std::endl;
                        int p = GetRand(10, 100); //
                        usleep(p * 1000);         // removing these lines solves the problem, but ... 
                    }
                    else
                    {
                        std::cout << "read error" << std::endl;
                        running = false;
                    }
                }
            }
        }
    }
    return nullptr;
}



int main(int argc, char *argv[])
{       
    if(argc < 2)
        return 1;

    srand(static_cast<unsigned int>(time(nullptr)));
    InitSocket();
    std::string app_type(argv[1]);
    if(app_type == "-s")
        InitServer();
    else if(app_type == "-c")
        InitClient();
    else
        return 1;

    running = true;
    pthread_create(&thread, nullptr, &ReadThread, nullptr);

    if(app_type == "-s")
    {
        while(polls[1].fd == 0) usleep(100);
    }

    char sendbuf[100];
    for(int i = 0;i < 100;i ++)
    {
        int length = GetRand(10, 100);
        int delay = GetRand(1, 10);
        for(int j = 0;j < length;j ++)
        {
            sendbuf[j] = GetRand('a', 'z');
        }

        unsigned long pending = 0;
        err = ioctl(polls[1].fd, TIOCOUTQ, &pending);
        std::cout << "pending " << pending << std::endl;

        err = send(polls[1].fd, sendbuf, length, MSG_NOSIGNAL);
        std::cout << "write " << length << " bytes" << std::endl;
        usleep(delay * 1000);
    }

    running = false;
    pthread_join(thread, nullptr);

    if(app_type == "-s")
        close(polls[0].fd);
    close(polls[1].fd);

    return 0;
}

running the code: the server:

test_app -s 

and then the client app (in another console window):

test_app -c 

A possible output:

The server:

client connected
         read 83 bytes: 
pending 0
write 66 bytes
pending 0
write 18 bytes
pending 768          <----- ????
write 17 bytes
pending 1536         <----- ????
write 79 bytes
         read 100 bytes: 
pending 2304         <----- ????
write 51 bytes
pending 3072
write 25 bytes
         read 100 bytes: 
pending 2304
write 96 bytes
pending 3072
write 19 bytes

The client:

pending 0
write 83 bytes
         read 66 bytes: 
pending 0
write 19 bytes
pending 768
write 94 bytes
pending 1536
write 52 bytes
pending 2304
write 11 bytes
pending 3072
write 39 bytes
pending 3072
write 35 bytes
         read 100 bytes: 
pending 3840
write 20 bytes
pending 2304
write 92 bytes
pending 3072
write 53 bytes
pending 3840
write 88 bytes
pending 4608
write 84 bytes
         read 100 bytes: 

As I noticed this problem occurs after adding an artificial delay in the read thread. In my real application I process a message and that creates a small delay. But I absolutely do not understand from where these numbers come (pending) and how the delay in the read thread affects this?

How can I avoid this? How can I get the real write buffer, not these strange numbers?

Answer 1

I use Unix domain sockets [... and] To prevent some overflow I check the write buffer

This is not your responsibility, nor, as far as I can determine, within your power. In particular, not all ioctls are applicable to all devices. TIOCOUTQ is an ioctl for terminal devices -- the leading 'T' in the macro name is mnemonic for this -- and it is not listed among the ioctls applicable to sockets in Linux (since you mention in comments that the target is Linux).

The kernel ensures that the buffer does not overflow. When there is a shortage of buffer space it will either block write requests until sufficient space becomes available or perform short writes or both. Or if the socket is in non-blocking mode then writes that would otherwise block will instead fail with EAGAIN .

How can I avoid this?

Avoid using ioctls on devices to which they do not apply.

How can I get the real write buffer, not these strange numbers?

You cannot, as far as I am aware, and it seems unlikely that you have a genuine need to do.