How to increase or keep starting speed of copying a file

Question

I am using these codes to copy a big file:

const int CopyBufferSize = 64 * 1024;

string src = @"F:\Test\src\Setup.exe";
string dst = @"F:\Test\dst\Setup.exe";

public void CopyFile()
{
        Stream input = File.OpenRead(src);
        long length = input.Length;
        byte[] buffer = new byte[CopyBufferSize];
        Stopwatch swTotal = Stopwatch.StartNew();

        Invoke((MethodInvoker)delegate
        {
            progressBar1.Maximum = (int)Math.Abs(length / CopyBufferSize) + 1;
        });


        using (Stream output = File.OpenWrite(dst))
        {
            int bytesRead = 1;
            // This will finish silently if we couldn't read "length" bytes.
            // An alternative would be to throw an exception
            while (length > 0 && bytesRead > 0)
            {
                bytesRead = input.Read(buffer, 0, Math.Min(CopyBufferSize, buffer.Length));
                output.Write(buffer, 0, bytesRead);
                length -= bytesRead;

                Invoke((MethodInvoker)delegate
                {
                    progressBar1.Value++;
                    label1.Text = (100 * progressBar1.Value / progressBar1.Maximum).ToString() + " %";
                    label3.Text = ((int)swTotal.Elapsed.TotalSeconds).ToString() + " Seconds";
                });
            }

            Invoke((MethodInvoker)delegate
            {
                progressBar1.Value = progressBar1.Maximum;
            });
        }

        Invoke((MethodInvoker)delegate
        {
            swTotal.Stop();
            Console.WriteLine("Total time: {0:N4} seconds.", swTotal.Elapsed.TotalSeconds);
            label3.Text += ((int)swTotal.Elapsed.TotalSeconds - int.Parse(label3.Text.Replace(" Seconds",""))).ToString() + " Seconds";
        });
    }

The file size is about 4 GB.

In the first 7 seconds it can copy up to 400 MB then this hot speed calms down.

What happen and how to keep this hot speed or even increase it?

Another question is here: When the file copied, windows is still working on destination file(about 10 seconds).

Copy Time: 116 seconds

extra time: 10-15 seconds or even more

How to remove or decrease this extra time?

Answer 1

What happens? Caching, mostly.

The OS pretends you copied 400 MiB in seven seconds, but you didn't. You just sent 400 MiB to the OS (or file system) to write in the future, and that's as much as the buffer can take. If you try to write a 400 MiB file and you pull the plug as soon as it's "done", your file will not be written. The same thing deals with the "overtime" - your application has sent all it has to the buffer, but the buffer isn't yet written to the drive itself (either its buffer, or even slower, the actual physical platter).

This is especially visible with USB flash drives, which tend to use caching heavily. This makes working with the (usually very slow) drive much more pleasant, with the trade-off that you have to wait for the OS to finish writing everything before pulling the drive out (that's why you get the "Safe remove" icon).

So it should be obvious that you can't really make the total time shorter. All you can do is try and make the user interface reflect reality a bit better, so that the user doesn't see the "first 400 MiB are so fast!" thing... but it doesn't really work well. In any case, your read->write speed is ~30 MiB/s. The OS just hides the peaks to make it easier to deal with the slow hard drive - very useful when you're dealing with lots of small files, worthless when dealing with files bigger than the buffer.

You have a bit of control over this when you use the FileStream constructor directly, instead of using File.OpenWrite - you can use FileOptions.WriteThrough to instruct the OS to avoid any caching and write directly to disk[1], giving you a better idea of the real write speed. Do note that this usually makes the total time larger, though, and it may make concurrent access even worse. You definitely don't want to use it for small files.

[1] - Haha, right. The drive usually has caching of its own, and some ignore the OS' pleas. Tough luck.

Answer 2

One thing you could try is to increase the buffer size. This really matters when the write cache can no longer keep up (as discussed in other answer). Writing a lot of small blocks is often slower than writing a few large blocks. Instead of 64 kB, try 1 MB, 4 MB or even bigger:

const int CopyBufferSize = 1 * 1024 * 1024;   // 1 MB
// or
const int CopyBufferSize = 4 * 1024 * 1024;   // 4 MB