c 中的读/写程序复制文件，我认为它复制的时间太长

Question

First code uses preset buffer and when i set buffer to 512byte and i need to copy 100MB file it takes about 1 second, but when I use 1byte buffer it takes over 3 minutes to copy 100MB file, on the other hand i have other code that uses fread and fwrite functions and it is about 0.5sec faster on 512byte buffer but it only takes him about 13 seconds to copy 100 mb file with 1byte buffer can someone see any error in code that uses system calls(read, write, open)

---

1. Code that uses(read, write...)

int main(int argc, char* argv[])
{
    char sourceName[20], destName[20], bufferStr[20];
    int f1, f2, fRead;
    int bufferSize = 0;
    char* buffer;

    bufferSize = atoi(argv[3]);

    buffer = (char*)calloc(bufferSize, sizeof(char));
    strcpy(sourceName, argv[1]);
   
    f1 = open(sourceName, O_RDONLY);
    if (f1 == -1)
        printf("something's wrong with oppening source file!\n");
    else
        printf("file opened!\n");

    strcpy(destName, argv[2]);
    f2 = open(destName, O_CREAT | O_WRONLY | O_TRUNC | O_APPEND);
    if (f2 == -1)
        printf("something's wrong with oppening destination file!\n");
    else
        printf("file2 opened!");

    fRead = read(f1, buffer, sizeof(char));
    while (fRead != 0)
    {
        write(f2, buffer, sizeof(char));
        fRead = read(f1, buffer, sizeof(char);
    }
    close(f1);
    close(f2);


    return 0;
}

---

2. Code that uses(fread, fwrite...)

int main(int argc, char* argv[]) {
    FILE* fsource, * fdestination;

    char sourceName[20], destinationName[20], bufferSize[20];
    //scanf("%s %s %s", sourceName, destinationName, bufferSize);
    strcpy(sourceName, argv[1]);
    strcpy(destinationName, argv[2]);
    strcpy(bufferSize, argv[3]);
    int bSize = atoi(bufferSize);
    printf("bSize = %d\n", bSize);
    fsource = fopen(sourceName, "r");
    if (fsource == NULL)
        printf("read file did not open\n");
    else
        printf("read file opened sucessfully!\n");
    
    fdestination = fopen(destinationName, "w");
    if (fdestination == NULL)
        printf("write file did not open\n");
    else
        printf("write file opened sucessfully!\n");

    char *buffer = (char*)calloc(bSize, sizeof(char));
    int flag;

    printf("size of buffer: %d", bSize);

    while (0 < (flag = fread(buffer, sizeof(char), bSize, fsource)))
        fwrite(buffer, sizeof(char), bSize, fdestination);
   

    fclose(fsource);
    fclose(fdestination);
    return 0;
}

EDIT: These are my measurements for buffers I took 20 measurements for each buffer and each file malaDat(1byte), srednjaDar(100MB), velikaDat(1GB)

Answer 1

Side note: sizeof(char) is always 1 by definition . So, just don't use sizeof(char) --it's frowned upon. And, I think it's adding to your confusion.

Because your example using read/write is using sizeof(char) as the count (the 3rd argument), it is only transferring one byte on each loop iteration (ie very slow).

At a guess, I think you're confusing the count for read/write with the size argument to fread/fwrite .

What you want is:

while (1) {
    fRead = read(f1, buffer, bufferSize);
    if (fRead <= 0)
        break;
    write(f2, buffer, fRead);
}

Also, fread/fwrite will [probably] pick an optimal buffer size [possibly 4096]. You can use setbuf to change that.

I've found [from some documentation somewhere] that for most filesystems [under linux, at least] the optimal transfer size is 64KB (ie 64 * 1024 ).

So, try setting bufferSize = 64 * 1024 .

IIRC, there is an ioctl/syscall that can return the value of the optimal size, but I forget what it is.

---

UPDATE:

Ok but when I choose 1byte buffer it is still too slow it takes more than 40mins to copy.

Of course, a 1 byte transfer [buffer] size will produce horrible results.

and when i use fread and fwrite with same buffer it takes way less time it takes about 3mins why is that?

How big is the file (ie what are the respective transfer rates [in MB/sec])? I'll assume your system can transfer at 10 MB/sec [conservative--30 MB/sec [minimum] for recently new system]. So, this is 600 MB/min and the file is approx 1.8 GB?

When you specify a 1 byte transfer/buffer size to read/write they do exactly what you tell them to do. Transfer 1 byte. So, you'll do ~2 billion read syscalls and 2 billion write syscalls!!!

syscalls are generally slow.

stdio streams have an internal buffer. This is set to an optimal size, let's say: 4096.

fread [and fwrite ] will fill/drain that buffer by calling [internally] read/write with a count of 4096.

So, fread/fwrite are doing 4096 times fewer syscalls. So, only about 470,000 syscalls. Quite a reduction.

The transfer to your buffer from the internal buffer is done a byte at a time, but that is only a short/fast memcpy operation done totally within the userspace application. This is much faster than issuing a syscall on a per byte basis.

So, the transfer size you pass to fread/fwrite does not affect the size of its internal buffer.

fread/fwrite only issue a read/write syscall to replenish/drain the stream's internal buffer when it is empty/full [respectively], regardless of what length you give in your fread/fwrite call.

If you want to slow down fread/fwrite , look at the man page for setbuf et. al. and do:

setbuffer(fsource,NULL,1);
setbuffer(fdestination,NULL,1);

---

UPDATE #2:

So this is totally normal? I am asking because this is my university task to do this measurements and my colleagues are getting results for system calls about 2 mins slower then user calls and i get much slower results

If you check with them, I'll bet they're using a larger buffer size.

Remember that your original read/write code had a bug that would only transfer a byte at a time, regardless of what you set bufferSize to [from the command line].

That's why I changed the loop in my original post.

To overachieve...

Look at O_DIRECT for open . If you use posix_memalign instead of malloc , you can force buffer alignment to be a multiple of the page size (4096) that allows O_DIRECT to work. And, setting a buffer size that is a multiple of the page size.

This option bypasses the read/write syscall's copy/transfer operation from your userspace from the kernel's internal page/filesystem cache and has the DMA H/W transfer directly to/from your buffer.

Also, consider adding O_NOATIME

Also, there is a linux syscall that is specifically designed to bypass all userspace memory/buffering to have the kernel do a file-to-file copy. It is sendfile , but it is similar to memcpy but uses file descriptors, an offset and a length.

And, the fastest way to access file data is to use mmap . See my answers:

1. How does mmap improve file reading speed?
2. read line by line in the most efficient way *platform specific*