使用Sharpziplib解壓縮24GB tar.gz

Question

我目前有一個提取tar.gz的進程，但當前進程無法處理超過4GB的內存...我想知道我做錯了什么，導致我使用sharpzip lib出錯了，它說參數長度不能小於零...。所有注釋掉的代碼都是當前過程

請看下面的代碼，並給我指示

public static void ExtractTarGZFiles(string strExtractionPath, string strInboundFolder)
    {
        List<string> files = new List<string>();
        string strFile = Path.GetFullPath(ConfigurationManager.AppSettings["InboundFolderPath"].ToString());
        if (mblnRunForFile)
        {
            System.IO.FileInfo ioInfo = new FileInfo(mstrFilename);
            strFile = ioInfo.DirectoryName;
            files.Add(ioInfo.Name);

        }
        else
        {
            files = System.IO.Directory.GetFiles(ConfigurationManager.AppSettings["InboundFolderPath"], "*.gz").ToList();
        }

        foreach (string file in files)
        {
            try
            {
            //string stFilePath = file;
            //string testingthisdamnthing = strInboundFolder + mstrFilename;
            //FileStream xstream = new FileStream(stFilePath, FileMode.Open, FileAccess.Read, FileShare.None);
            //xstream.Close();
            //string strTemp = ConfigurationManager.AppSettings["ExtractTempFolderPath"];
            //TarArchive objTA = TarArchive.CreateInputTarArchive(new GZipStream(new FileStream(stFilePath, FileMode.Open, FileAccess.Read), CompressionMode.Decompress));
            //objTA.ProgressMessageEvent += ExtractTarNotifier;
             //objTA.ExtractContents(strTemp);



            //DirectoryInfo dirtemp = new DirectoryInfo(strTemp);
            DirectoryInfo dirExtract = new DirectoryInfo(strExtractionPath);

            Stream inStream = File.OpenRead(stFilePath);
            Stream gzipStream = new GZipInputStream(inStream);

            TarArchive tarArchive = TarArchive.CreateInputTarArchive(gzipStream);
            tarArchive.ExtractContents(ConfigurationManager.AppSettings["ExtractFolderPath"]);
            tarArchive.Close();

            gzipStream.Close();
            inStream.Close();
            //while (IsFileExistsinTempPath(dirtemp, dirExtract))
            //{
            //    //Do nothing
            //}
            //CopyFilesFromTempToExtract(dirtemp, dirExtract);
            //objTA.Close();
            //Logger.Write(" Tar.Gz files Decompressed Successfully");
            MonthLog.Log("Tar.Gz files Decompressed Successfully", "Month", 3, 2, System.Diagnostics.TraceEventType.Information, mstrFilename);
            System.IO.File.Copy(stFilePath, ConfigurationManager.AppSettings["ArchiveFolderPath"] + new FileInfo(stFilePath).Name, true);
            File.Delete(stFilePath);
            //Logger.Write(" Tar.GZ files Moved to Archive Folder");
            MonthlyGreenPackageLog.Log("Tar.Gz files Moved to Archive Folder", "Month", 3, 2, System.Diagnostics.TraceEventType.Information, mstrFilename);

            }
                catch (System.IO.IOException ex)
                {
                    //go to next file
                    //Logger.Write("Unable to open compressed file");
                    MonthLog.Log("Unable to open compressed file", "Month", 1, 1, System.Diagnostics.TraceEventType.Error, mstrFilename);
                    Email objEmail1 = new Email();
                    objEmail1.IsBodyHTML = true;
                    objEmail1.FromAddress = ConfigurationManager.AppSettings["FromAddress"];
                    string[] strToAddresses = ConfigurationManager.AppSettings["ExceptionAddress"].Split(',');
                    objEmail1.SetToAddress(strToAddresses);
                    objEmail1.Subject = "The Month  File  " + mstrFilename + " Failed to Decompress ";
                    objEmail1.Body = " Exception " + ex.Message + " occured while decompressing file";
                    //objEmail.AddAttachment("Exception occured while processingfiles");
                    objEmail1.SendEmail();
                    //Logger.Write("Sent a mail to all the Address");
                    throw ex;
                }
                catch (Exception ex)
                {
                    //Logger.Write("Exception " + ex.Message + " occured while decompressing file");
                    MonthlyGreenPackageLog.Log("Exception " + ex.Message + " occured while decompressing file", "Monthl", 1, 1, System.Diagnostics.TraceEventType.Error, mstrFilename);
                    Email objEmail = new Email();
                    objEmail.IsBodyHTML = true;
                    objEmail.FromAddress = ConfigurationManager.AppSettings["FromAddress"];
                    string[] strToAddresses = ConfigurationManager.AppSettings["ExceptionAddress"].Split(',');
                    objEmail.SetToAddress(strToAddresses);
                    objEmail.Subject = "The tar.gz Month File  " + mstrFilename + " Failed to Decompress ";
                    objEmail.Body = " Exception " + ex.Message + " occured while decompressing file";
                    //objEmail.AddAttachment("Exception occured while processing ADX files");
                    objEmail.SendEmail();
                    //Logger.Write("Sent a mail to all the Address");
                    throw ex;
                }

            }
        }

Answer 1

如果您的應用程序為x86，則每個進程只能有4GB的內存。 編譯為x64以使用更多資源。

Answer 2

這來自您正在使用的ZipLibrary。 您需要將應用程序編譯為x64程序，並在64位環境中運行它。

    // To permit the zip to be unpacked by built-in extractor in WinXP and Server2003, WinZip 8, Java, and other older code,
    // you need to do one of the following: Specify UseZip64.Off, or set the Size.
    // If the file may be bigger than 4GB, or you do not need WinXP built-in compatibility, you do not need either,
    // but the zip will be in Zip64 format which not all utilities can understand.

Answer 3

我為您提供了三種潛在的解決方案，但是我無法告訴您它們中的任何一種是否切實可行。

1. 使用“ 從流中解壓縮”示例通過內存流式傳輸它，因此您無需加載整個內容（這就是為什么現在失敗）的原因。
2. 切換到使用DotNetZip ，它可能具有更多選項來處理x86機器上的大文件。
3. 查找可以編寫腳本或可以監視目錄的常規解壓縮應用程序，並在將其解壓縮到應用程序之前對其進行解壓縮。