Error with R function download.file: Too many open files

Question

I am using:

 download.file(url,path_file,mode="wb",quiet=quiet)

with R version 3.2.3 (2015-12-10) on Windows 7 to copy a big amount of images (TIFF files). I have to copy a huge number of files: 300'00 but it failed at some point with the following issue:

"cannot open destfile 'tmp/74114070005_531__0.tiff' , reason 'Too many open files'"

The issue is that from time to time the copy of the url failed and R create a empty file with a size of 0 Byte but Windows lock the file so I cannot remove it. So the failed file stay open until I exit R. After a certain number of failed copy then I got the error above that "too many open file"

Is there a way to close the connection for each file ? I tried closeAllConnections() but this has not impact.

Is there a way to run a R command line that will "restart" the R programm so Windows will unlock the files ?

Any other idea are welcome.

Thanks

Fabien

Answer 1

This is an issue with Windows that locked files. The best things is to avoid Windows. For people that need to use Windows there is a solution. I tested it with Windows 7:

• copy the handle.exe (or handle64.exe) from: https://technet.microsoft.com/en-ca/sysinternals/bb896655.aspx?f=255&MSPPError=-2147217396
• open a cmd prompt windwos as administrator
• find the handle of the lock files: handle64.exe -nobanner -p python.exe pattern_of_your_file
• handle64.exe -p pid_your_file -c handle_your_file

This avoid to have all these locked files