C# FTP download files slow

Question

I have a question regarding the ftp library from C#. I need to download 9000 txt files from a ftp server. Station.ToUpper() is the file name, so for every file I need a new ftp connection. For one file it takes around one second. The txt files contain two lines. So for all files it takes around one and a half hour. Is there a better / faster solution?

            // Get the object used to communicate with the server.
            FtpWebRequest request = (FtpWebRequest)WebRequest.Create(ftpAddress + station.ToUpper());
            //request.UsePassive = false;
            request.Method = WebRequestMethods.Ftp.DownloadFile;

            // This example assumes the FTP site uses anonymous logon.
            request.Credentials = new NetworkCredential("anonymous", "janeDoe@contoso.com");

            FtpWebResponse response = (FtpWebResponse)request.GetResponse();

            Stream responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);

Answer 1

There's not much you're doing wrong in this code, except for the fact you're not calling Dispose() on your streams or response objects. Do that first to make sure you're not somehow running out of resources on the client or something.

Other than that, you don't have too many options here, and a lot depends upon what you can do on the server side.

First, you could try to use threading to download a bunch of files at once. You'll need to experiment with how this affects your throughput. It will probably scale linearly for a while, then fall off. If you open up too many connections, you could anger the maintainer of the server, or it could start denying you connections. Be conservative.

Optimally, the files would be zipped (.ZIP or .TGZ) on the server. This will likely not be an option if you don't have more control over the process.

Answer 2

Use the MGET command to avoid re-establishing a connection each time. The System.Net client does not support MGET , so you would have to use a third party library or script ftp.exe . Regardless of the client you choose, the FTP log would look like the following:

USER anonymous
PASS janeDoe@contoso.com
CWD path/to/file
// to get 3 named files
MGET file1.txt file2.txt file3.txt
// or to get all files matching a pattern
MGET *.txt

The file transfers will use the same control session, avoiding login and other network overhead.

One library which could be of interest is FTPLib , which avoids tearing down the channel on each command. Be careful, however, as FTPLib is based on wininet which is not allowable for use in an NT Service.

I would also take a look at LumiSoft , an open source project with a friendly license, and DotNetFtpLib , though I have used neither and cannot speak to their stability or featureset. On the scripting side, take a look at "Using FTP Batch Scripts" .