Can write() to a non-blocking fd return EAGAIN when select reports it as writable?

Question

I'm trying to track down some strange behavior in OS X (10.8.2). Basically, I'm opening a pipe, and filling it with data until it is unwritable. I'm finding, however, that depending on the size of the chunk I attempt to write, sometime I will get EAGAIN from a write() call even though select claims the pipe is still writable. Here is some test code:

#include <unistd.h>
#include <errno.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/select.h>

#define START 1
#define END 16

int is_writeable(int fd) {
    struct timeval timeout;
    timeout.tv_sec = 0;
    timeout.tv_usec = 0;

    fd_set ws;

    FD_ZERO(&ws);
    FD_SET(fd, &ws);

    if(select(fd+1, NULL, &ws, NULL, &timeout) < 0) {
        return -1;
    }

    if(FD_ISSET(fd, &ws))
        return 1;
    else
        return 0;
}

int main(int argc, char *argv[]) {

    int pipes[2];
    int rp, wp, i, errval, fails, tmp;

    char testbuf[END];
    char destbuf[128000];

    for(i = START; i < END; i++) {
        int byte_count = 0;
        printf("%i: ", i);
        fails = 0;

        pipes[0] = 0;
        pipes[1] = 0;

        if(pipe(pipes) < 0) {
            printf("PIPE FAIL\n");
            break;
        }
        rp = pipes[0];
        wp = pipes[1];

        int flags = fcntl(wp, F_GETFL, 0);
        if(fcntl(wp, F_SETFL, flags | O_NONBLOCK) == -1) {
            fails = 4;
        }

        if(is_writeable(wp) != 1) {
            fails = 1;
        }

        while(!fails) {
            // printf(".");
            if(is_writeable(wp) < 1) {
                break;
            }

            tmp = write(wp, testbuf, i);
            //No bytes written, fail
            if(tmp < 0) {
                if(errno == EAGAIN) {
                    if(is_writeable(wp) == 1) {
                        fails = 3;
                        break;
                    }
                } else {
                    fails = 2;
                    perror("During write");

                    break;
                }

            } else {
                byte_count += tmp;
            }
            //Errno is eagain, fail
        }
        printf("byte count %i, ", byte_count);

        if(fails)
            printf("FAIL, %i\n", fails);
        else
            printf("PASS\n");

        if(close(wp) != 0)
            printf("WP CLOSE FAIL\n");
        if(close(rp) != 0)
            printf("RP CLOSE FAIL\n");
    }

}

Here is the output:

1: byte count 16384, PASS
2: byte count 16384, PASS
3: byte count 65535, FAIL 3
4: byte count 16384, PASS
5: byte count 65535, FAIL 3
6: byte count 65532, FAIL 3
7: byte count 65534, FAIL 3
8: byte count 16384, PASS
9: byte count 65529, FAIL 3
10: byte count 65530, FAIL 3
11: byte count 65527, FAIL 3
12: byte count 65532, FAIL 3
13: byte count 65533, FAIL 3
14: byte count 65534, FAIL 3
15: byte count 65535, FAIL 3

Note that failure case 3 is where a write() call returns -1, but select still reports that the filehandle is writable.

Here is a shorter example in Ruby that shows the same failure:

(1..10).each do |i|
  passes = true
  begin
    (rp, wp) = IO.pipe
    wp.write_nonblock ("F" * i) while(select [], [wp], [], 0)
  rescue Errno::EAGAIN
    puts "#{i}: FAIL"
    passes = false
  ensure
    rp.close
    wp.close
  end
  puts "#{i}: PASS" if passes
end

I can't tell for certain if this is a bug or a misinterpretation of the specification. Thoughts?

Answer 1

You're using pipes here. Pipes have a fun atomic write property --- writes smaller than PIPE_BUF (4096 here) bytes are guaranteed to be atomic. So a write to a pipe can fail with EAGAIN even if it's possible to write a smaller number of bytes to the pipe.

I'm not sure if that's what you're running into here (I haven't looked too closely), but this behaviour is documented in man 7 pipe.