如何在C＃中以更好的方式提取和处理文件

Question

I have around 5000 files stored in a FTP location.So first i have to download file from FTP ,then i have to un zip the .gz file and finally i have to process the file and push the data to oracle database.i used TamirSSh assembly to retrieve from FTP and ionic.zip to unzip file. But Downloadfile() ,Extractfile() and ProcessFile() methods taking long time to finish.What would be the better way to download ,unzip and process file in c#.This is a console application

static void Main(string[] args)
    {
        Downloadfile();
    }

private static void Downloadfile()
{
     //Download 5000 file
       Sftp ftp = new Sftp(dtr["FTP_SERVER"].ToString(), dtr["FTP_USER_ID"].ToString(), dtr["FTP_PASSWORD"].ToString());
                                    ftp.Connect<ftp://ftp.connect/>();
                                    System.IO.Directory.CreateDirectory(@localDestnDir);
                                    ArrayList list;
                                    list = ftp.GetFileList(remotepath<ftp://ftp.getfilelist(remotepath/>);
                                    //GExport_EI_DN_G_6542_StarMetroDeiraHotel&Apartment_10.235.155.37_20161120003108.xml.gz
                                    foreach (string item in list)
                                    {
                                        if (item.StartsWith("GExport_") &&(!item.ToUpper().Contains("DUM")))
                                        {
                                            path = item;
                                            //path = "GExport_EI_DN_G_6542_StarMetroDeiraHotel&Apartment_10.235.155.37_20161120003108.xml.gz";
                                            ftp.Get(dtr["REMOTE_FILE_PATH"].ToString() + path, @localDestnDir + "\\" + dtr["SOURCE_PATH"].ToString());
                                            download_location_hw = dtr["LOCAL_FILE_PATH"].ToString();
                                          //  ExtractZipfiles(download_location_hw + "//" + path, dtr["REMOTE_FILE_PATH"].ToString(), dtr["FTP_SERVER"].ToString(), dtr["FTP_USER_ID"].ToString(), dtr["TECH_CODE"].ToString(), dtr["VENDOR_CODE"].ToString());
                                        }
                                    }
                                    ftp.Close();
        //extract 5000 file by using Ionic.zip 
                    Extractfile();
        //then process 5000 files
        ProcessFile();
}

But Downloadfile() ,Extractfile() and ProcessFile() methods taking long time to finish.What would be the better way to download ,unzip and process file in c#.This is a console application

Answer 1

Basically the pipleline download compressed file, extract and process it is fine. But while your system is processing it could download the next ones in parallel because network transport is not CPU intensive compared to decompressing and processing.

One very simple and fast approach is to use Parallel.ForEach in your loop and ExtractFile and ProcessFile in the loop as well. To sketch this idea:

private static void Downloadfile()
{
    //Download 5000 file
    Sftp ftp = new Sftp(dtr["FTP_SERVER"].ToString(), dtr["FTP_USER_ID"].ToString(), dtr["FTP_PASSWORD"].ToString());
    ftp.Connect<ftp://ftp.connect/>();
    System.IO.Directory.CreateDirectory(@localDestnDir);       
    var list = ftp.GetFileList(remotepath).ToList();

    Parallel.ForEach(list, item => 
        {
            if (item.StartsWith("GExport_") &&(!item.ToUpper().Contains("DUM")))
            {
                path = item;
                //path = "GExport_EI_DN_G_6542_StarMetroDeiraHotel&Apartment_10.235.155.37_20161120003108.xml.gz";
                ftp.Get(dtr["REMOTE_FILE_PATH"].ToString() + path, @localDestnDir + "\\" + dtr["SOURCE_PATH"].ToString());
                download_location_hw = dtr["LOCAL_FILE_PATH"].ToString();
                //  ExtractZipfiles(download_location_hw + "//" + path, dtr["REMOTE_FILE_PATH"].ToString(), dtr["FTP_SERVER"].ToString(), dtr["FTP_USER_ID"].ToString(), dtr["TECH_CODE"].ToString(), dtr["VENDOR_CODE"].ToString());
            }

            //extract file by using Ionic.zip 
             Extractfile(item);   <= Extractfile works on a single file now
            //then process file
            ProcessFile(item);    <= ProcessFile works on a single file now
        });
        ftp.Close();

}

Answer 2

Without seeing all your code it is hard to say, but most likely you could benefit from parallelization. This is now wonderfully easy to do in C#. Instead of that foreach loop you are currently using, try something like this:

        Parallel.ForEach(list.ToArray(), item => {
            // Download the item with ftp.Get
            // Unzip the file you just downloaded
            // Process the file
        });

The speed benefit of this is that you will be doing the off-line processing of the first files (unzipping, processing) while the computer is also waiting to download the next files.

Now, this will try to download several files at once. That might not be a good idea, because you might overwhelm the FTP server. So another way to do it is download the files one at a time, and then immediately process each in the background while the foreground moves on to downloading another file:

        Task[] myTasks = new Task[list.Count];
        int i = 0;
        foreach (string item in list)
        {
            // Download the item with ftp.Get and store its location in download_location_hw
            ftp.Get(dtr["REMOTE_FILE_PATH"].ToString() + path, @localDestnDir + "\\" + dtr["SOURCE_PATH"].ToString());
            string download_location_hw = dtr["LOCAL_FILE_PATH"].ToString();
            // Spin off a background task to process the file we just downloaded
            myTasks[i++] = Task.Run(() => {
                // Extract the zip file referred to by  download_location_hw
                // Process the extracted zip file
            });
        }
        Task.WaitAll(myTasks);

For both examples make sure you are using System.Threading.Tasks;