io.Copy is a lot slower than calling the mv command

Question

It seems io.Copy is slow for me:

   _,err = io.Copy(destf,srcf)

io.Copy takes longer, about 2 minutes to copy a 1GB file to a network share. Noticed that mv.exe finishes the job in ~25 seconds max - so I've started to evoke mv for my jobs.

    output, err := exec.Command("mv", src, dest_folder).CombinedOutput()

This slowness is consistently reproducible on my end, Any tips on how to speeden will be much appreciated!

Update:

Thanks for the suggestion to use io.CopyBuffer() , however mv.exe still emerges as the sole victor, by a respectable margin.

Details:

PS C:\temp> .\move_files.exe .\testfile.data "\\somehost\somefolder\bleh13.txt"
2018/07/14 19:04:54 Created C:\Temp\2\deleteME__913153343, copy of .\testfile.data, Size: 1073745920 bytes
2018/07/14 19:05:55 Transfer with io.Copy() took us 60.836702 seconds
2018/07/14 19:06:47 Transfer with io.CopyBuffer() took us 50.729625 seconds
2018/07/14 19:06:59 Transfer with mv command took us 11.470456 seconds
PS C:\temp>

You're welcome to try this yourself : https://play.golang.org/p/2_lR83A4BXe

Answer 1

You compare copy and move which are different things. Moving a file doesn't assume writing a copy of all data into the new physical memory destination, just rewriting some headers/descriptors in the file system. So for proper comparison benchmark

• os.Rename(oldpath, newpath string) vs mv.exe
• io.Copy vs cp.exe

Answer 2

Maybe try io.CopyBuffer : io.Copy uses a buffer of 32*1024 bytes .

Check if a bigger or smaller buffer size makes a difference.

Answer 3

Continuing my comment suggesting to try the CopyFile Win32 API function, here's a working program which wraps it:

package main

import (
    "log"
    "os"
    "syscall"
    "unsafe"
)

var (
    kernel32     = syscall.MustLoadDLL("kernel32.dll")
    copyFileProc = kernel32.MustFindProc("CopyFileW")
)

func CopyFile(src, dst string, overwrite bool) error {
    srcW := syscall.StringToUTF16(src)
    dstW := syscall.StringToUTF16(dst)

    var failIfExists uintptr
    if overwrite {
        failIfExists = 0
    } else {
        failIfExists = 1
    }

    rc, _, err := copyFileProc.Call(
        uintptr(unsafe.Pointer(&srcW[0])),
        uintptr(unsafe.Pointer(&dstW[0])),
        failIfExists)
    if rc == 0 {
        return &os.PathError{
            Op:   "CopyFile",
            Path: src,
            Err:  err,
        }
    }
    return nil
}

func main() {
    log.SetFlags(0)

    if len(os.Args) != 3 {
        log.Fatalf("Wrong # args.\nUsage: %s SOURCE DEST\n", os.Args[0])
    }

    err := CopyFile(os.Args[1], os.Args[2], false)
    if err != nil {
        log.Fatal("error copying file: ", err)
    }
}

Please try it and see whether it improves over io.Copy* .