Read file using libaio with O_DIRECT flag

Question

I am reading file using asynchronous libaio. The code below shows how I do that. The code works fine, but now I want to switch to O_DIRECT mode to avoid file caching. When I change the line 25 to fd = open("./testfile", O_RDONLY|O_DIRECT); the program stops working correctly. (the io_getevents returns no data). Could you please help me to adjust the program so that it works correctly also with O_DIRECT flag?

Thanks in advance

OS: Ubuntu 12.10 3.5.0-26-generic)

1 #include <unistd.h>
2 #include <stdio.h>
3 #include <stdlib.h>
4 #include <sys/types.h>
5 #include <sys/stat.h>
6 #include <sys/param.h>
7 #include <fcntl.h>
8 #include <errno.h>
9 #include <libaio.h>
10 
11 int main(int argc, char* argv[]) {
12 
13   struct iocb cb;
14   struct iocb* iocbs = &cb;
15   struct io_event events[1];
16   char data[4096];
17   io_context_t ctx;
18   int fd;
19   int res;
20 
21   memset(&ctx, 0, sizeof(ctx));
22   memset(&cb, 0, sizeof(cb));
23   memset(&data, 0, sizeof(data));
24 
25   fd = open("./testfile", O_RDONLY);
26 
27   if(io_setup(1, &ctx) < 0) {
28     printf("io_setup error\n");
29     exit(-1);
30   }
31   printf("io_setup OK\n");
32 
33   // submit read request
34   io_prep_pread(&cb, fd, &data, 1024, 0);
35   res = io_submit(ctx, 1, &iocbs);
36   if(res < 0) {
37     printf("io_submit error\n");
38     exit(-2);
39   }
40   printf("io_submit OK: %d\n", res);
41 
42   // get events
43   res = io_getevents(ctx, 0, 1, events, NULL);
44   if(res < 0) {
44   if(res < 0) {
45     printf("io_getevents Error: %d", res);
46     exit(-3);
47   }
48   printf("io_getevents OK: %d %li %li\n", res, events[0].res, events[0].res2);
49 
50   // dump data received
51   char data_prefix[16];
52   strncpy(data_prefix, data, 15);
53   data_prefix[15] = 0;
54   printf("data: %s\n", data_prefix);
55 
56   res = io_destroy(ctx);
57   if(res < 0) {
58     printf("io_destroy Error: %d\n", res);
59     exit(-4);
60   }
61   printf("io_destroy OK: %d\n", res);
62 
63   res = close(fd);
64   printf("close: %d\n", res);
65 }

Answer 1

You need to align the buffer to 512 bytes for O_DIRECT to work correctly. Before io_prep_pread :

char *data;
posix_memalign(&data, 512, 1024);

Note that the size read has to be a multiple of 512.

Answer 2

Change char *data; to void* data; char *data; to void* data;

then do posix_memalign(&data,512,1024);

Answer 3

There are two main issues with your example:

---

First, you call io_getevents() with min_nr set to zero:

43   res = io_getevents(ctx, 0, 1, events, NULL);

And thus its perfectly fine for that function to return 0 read events if you call it before the read finished.

See also io_getevents(2) :

The io_getevents() system call attempts to read at least min_nr events and up to nr events from the completion queue of the AIO context speci‐ fied by ctx_id.

If you set the min_nr argument to 1, ie call io_getevents(ctx, 1, 1, ...) then the function blocks in case the read hasn't finished yet - until it is finished.

---

Second, your read buffer isn't necessarily correctly aligned for O_DIRECT and the read size might not be dividable by the sector size of the device - cf. open(2) :

The O_DIRECT flag may impose alignment restrictions on the length and address of user-space buffers and the file offset of I/Os. In Linux alignment restrictions vary by filesystem and kernel version and might be absent entirely. However there is currently no filesystem-indepen‐ dent interface for an application to discover these restrictions for a given file or filesystem. Some filesystems provide their own inter‐ faces for doing so, for example the XFS_IOC_DIOINFO operation in xfsctl(3).

Under Linux 2.4, transfer sizes, and the alignment of the user buffer and the file offset must all be multiples of the logical block size of the filesystem. Since Linux 2.6.0, alignment to the logical block size of the underlying storage (typically 512 bytes) suffices. The logical block size can be determined using the ioctl(2) BLKSSZGET operation or from the shell using the command:

 blockdev --getss

Thus, it really depends on your kernel version, filesystem and drive what alignment you need and whether the stack allocated data array is aligned as required, by chance.

Something like this should be sufficient to fix the alignment and size constraint violations (unless your device has even greater physical sectors):

    void *data = 0;
    int r = posix_memalign(&data, 4096, 4096);
    if (r) {
        fprintf(stderr, "posix_memalign failed: %s\n", strerror(r));
        return 1;
    }

Note that if you change the allocation like this you have to adjust the related memset() call and other usages, as well.

---

Other issues with your example:

Your error handling is incomplete. That means you don't always check for errors. And in the places you do and report them you don't translate the error code to something readable (eg by passing the code through strerror() where appropriate).

Also, there is a line duplicated that introduces a syntax error:

44   if(res < 0) {
44   if(res < 0) {

The exit status of your program is undefined when it succeeds (ie return 0 missing at the end) and returning with a negative exit status is problematic.

When you print io_event::res and res2 you are using the wrong format specifier, correct is %lu ;