liburing / IORING_OP_PROVIDE_BUFFERS - 分配给读取操作的错误缓冲区

Question

I am playing with liburing and io_uring, the former is built from the master branch and for the latter i am using the kernel 5.8.0-rc3.

I am facing the issue I am going to explain on previous version of the kernel as well (5.7.6, 5.8.0-rc1).

The tests using the provide buffers look like working as expected.

I am trying to implement a network tcp/ip server using IORING_FEAT_FAST_POLL and IORING_OP_PROVIDE_BUFFERS and I am using https://github.com/frevib/io_uring-echo-server as code reference.

The issue I am facing is that the buffer id is always set to the last of the registered buffers but I can see that the actual buffer used by the kernel is not that.

I have reduced the amount of registered buffers to just 10 and added some debugging info, it's now printing the content of all the buffers

Upon the first RECV from the server I get in cqe->flags >> 16 the ID of the buffer containing the data and it's the last one of the list, and I can see that it's containing the data but upon the second RECV the kernel now used the buffer id 0 (the first one) but cqe->flags >> 16 is still containing the ID of the last one.

The cause of this is an invocation of io_uring_prep_provide_buffers , infact upon the second invocation the issues arise.

Out of curiosity I have also tried to register the buffers one by one using

    for(uint16_t i = 0; i<MAX_CONNECTIONS; i++) {
        sqe = io_uring_get_sqe(&ring);
        io_uring_prep_provide_buffers(sqe, bufs, MAX_MESSAGE_LEN, 1, group_id, i);
    }

instead of

    io_uring_prep_provide_buffers(sqe, bufs, MAX_MESSAGE_LEN, MAX_CONNECTIONS, group_id, 0);

The issue start immediately.

I have also entirely disabled the re-registration of the freed buffer and everything works as expected (I commented out the invocation of the function add_provide_buf ).

From the documentation it looks like after the buffer is selected the expectation is to add it back if it's free to be used but clearly this is causing the issue.

Any hint?

Thanks!

Answer 1

There was a bug in the code with re-registering the buffer: https://github.com/frevib/io_uring-echo-server/commit/aa6f2a09ca14c6aa17779a22343b9e7d4b3c7994

Checkout the latest master branch, it should work now: https://github.com/frevib/io_uring-echo-server