How can I copy a large file to/from a network share in Windows without inflating the file cache? PowerShell or .NET, ideally

Question

I am working with dedicated Windows 2008 R2 and 2003 SQL Servers; throughout the day and night I need to copy large files off the machines to network shares for backup and log shipping.

There's a documented issue where the OS will allocate a lot of memory to the file cache to handle these copies, and that robs memory, potentially, from other functions. (Eventually on Win2k8 R2, Task Manager reports Free Memory = 0 as the result of this behavior.) But these particular files are never read after the copy, and the "just in case" caching of the file data doesn't make sense for this function.

Is there a simple .NET or PowerShell way to copy files and not have this file cache grow? The biggest file I am dealing with, for reference, is 300+ GB.

We currently use RoboCopy on 2008 R2, xcopy on 2003, and SQL Server's own Log Shipping file copy job for this function.

I have heard about a /J switch in XCOPY in win2k8R2, and a special DLL from Exchange that some argue helps this issue but it would be nice to have a really clean and simple solution.

See also http://blogs.msdn.com/b/ntdebugging/archive/2007/11/27/too-much-cache.aspx

Answer 1

When I copy large amounts of data, I use Robocopy because I have run into problems with other options, eg XCOPY, Copy-Item, etc.

My Robocopy script:

#Source folder
$src = "" 

#Destination folder
$dest = "" 

<#Robocopy options: 
E = Copy subdirectories, including empty directories.
V = Verbose output
ZB = Restartable & backup mode (to handle network connectivity issues & disruptions)
COPY DAT = Copy Data, Attributes, and Timestamps. Add 'S' to include ACL/permissions.
XJ = Exclude junction points.
#>
$options = @("/E","/V","/ZB", "/COPY:DAT", "/XJ")

#Store src, dest, and other previously defined options in an array.
$cmdArgs = @($src, $dest, $options)

#Execute!
robocopy @cmdArgs