文件系统性能非常差

Question

I have a function that basically does this:

int mmkdir(const char *path, mode_t mode)
{
    struct stat st;
    if (stat(path, &st) < 0)
    {
        if (errno != ENOENT)
            return -1;

        if (mkdir(path, S_IWUSR | S_IRUSR | S_IXUSR | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH) < 0 && errno != EEXIST)
            return -1;
    }
    else if (!S_ISDIR(st.st_mode))
    {
        errno = ENOTDIR;
        return -1;
    }
    return 0;
}

int fun(int id, int id2)
{
    time_t ts;
    struct tm timeinfo;
    char buff[1024], buff1[20], buff2[20], buff3[20];

    ts = time(NULL);
    localtime_r(&ts, &timeinfo);
    strftime(buff1, sizeof(buff1), "%Y%m%d", &timeinfo);
    strftime(buff2, sizeof(buff2), "%H", &timeinfo);

    snprintf(buff, sizeof(buff), "%s/%s", dir, buff1);
    if (mmkdir(buff, ORDER_FILE_DEFAULT_PERMS) < 0)
        return -1;

    len = strlen(buff);
    snprintf(buff + len, sizeof(buff) - len, "/%s", buff2);
    if (mmkdir(buff, ORDER_FILE_DEFAULT_PERMS) < 0)
        return -1;
    len += strlen(buff2) + 1;

    strftime(buff3, sizeof(buff2), "%M%S", &timeinfo);
    snprintf(buff + len, sizeof(buff) - len - 1, "/%010d-%04d-%s%s.%s",
         id, id2, buff2, buff3, ext);

    return open(buff, O_WRONLY | O_APPEND | O_CREAT, S_IWUSR | S_IRUSR | S_IRGRP | S_IROTH);
}

Plese don't check for errors, it's a working code. It basically calls stat (2), mkdir (2) (sometimes) and open (2).

The problem is that when to I/O load on the server is pretty high, this piece of code sometimes takes even 7s (!!) to complete.

These files that this functions create are located in a folder in /, which is mounted:

/dev/md0 on / type ext4 (rw,errors=remount-ro)

and can be up to 1000 files in a single folder.

What may be the problem? Why this may take so long? Is there any misconfiguration?

I don't ask only for improvements in the code, but in the server configuration also, if possible.

As per request in the comments, the output of cat /proc/mdstat is:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md3 : active raid1 sdd1[1] sdc1[0]
      488282000 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      15624120 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[0] sdb3[1]
      1958900 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      470798200 blocks super 1.2 [2/2] [UU]

that means that the disks are ok

Answer 1

If this function is called substantially more than once per hour, and the folders are not deleted, then your mmkdir calls are redundant. I would implement some kind of caching scheme where you remember the last hour in which you created a folder, and skip creating a new folder if you already created it. This will remove two stat calls, which could make a big difference on an overloaded system.

There's likely to be no further improvement you can make to the code alone, since the open call would be the only remaining system call, and it cannot be elided without changing the meaning of the function.