[英]c# working on excel files with large data
I'm copying data from first sheet of different excel files to a single workbook. 我正在将数据从第一张不同的excel文件复制到单个工作簿。 I already have tried it with different alternatives like
npoi
, spire.xls
and Interop
which works good, but it kills too much of my time. 我已经用
npoi
, spire.xls
和Interop
等其他替代方法进行了尝试,效果很好,但是却浪费了我太多的时间。 It would really be thankful if anyone can suggest me with a better one. 如果有人可以给我更好的建议,那将是非常感谢。 Been through many forms on the web, but couldn't find.
通过网络上的许多表格,但找不到。
FYI: Each of My files are more than 50 MB in size.
仅供参考:“我的每个文件”的大小均超过50 MB。 A few being 10 MB or less.
少数为10 MB或更少。
This is one of which I have tried (Uses Spire.xls): 这是我尝试过的方法之一(使用Spire.xls):
workbook = new Workbook();
//laod first file
workbook.LoadFromFile(names[0]);
//load the remaining files starting with second file
for (int i = 1; i < cnt; i++)
{
LoadFIle(names[i]);
//merge the loaded file immediately and than load next file
MergeData();
}
private void LoadFIle(string filePath)
{
//load other workbooks starting with 2nd workbbook
tempbook = new Workbook();
tempbook.LoadFromFile(filePath);
}
private void MergeData()
{
try
{
int c1 = workbook.ActiveSheet.LastRow, c2 = tempbook.Worksheets[0].LastRow;
//check if you have exceeded 1st sheet limit
if ((c1 + c2) <= 1048575)
{
//import the second workbook's worksheet into the first workbook using a datatable
//load 1st sheet of tempbook into sheet
Worksheet sheet = tempbook.Worksheets[0];
//copy data from sheet into a datatable
DataTable dataTable = sheet.ExportDataTable();
//load sheet1
Worksheet sheet1 = workbook.Worksheets[workbook.ActiveSheetIndex];
sheet1.InsertDataTable(dataTable, false, sheet1.LastRow + 1, 1);
}
else if ((c1 >= 1048575 && c2 >= 1048575) || c1 >= 1048575 || c2 >= 1048575 || (c1 + c2) >= 1048575)
{
workbook.Worksheets.AddCopy(tempbook.Worksheets[0]);
indx = workbook.ActiveSheet.Index;
workbook.ActiveSheetIndex = ++indx;
}
else
{
//import the second workbook's worksheet into the first workbook using a datatable
//load 1st sheet of tempbook into sheet
Worksheet sheet = tempbook.Worksheets[0];
//copy data from sheet into a datatable
DataTable dataTable = sheet.ExportDataTable();
//load sheet1
Worksheet sheet1 = workbook.Worksheets[workbook.ActiveSheetIndex];
sheet1.InsertDataTable(dataTable, false, sheet1.LastRow + 1, 1);
}
}
catch (IndexOutOfRangeException)
{
}
}
}
Well, this works good but as said takes a long time. 好吧,这很好用,但是需要花费很长时间。 Any suggestions are welcome.
欢迎任何建议。 Thanks in advance.
提前致谢。
Here is my (fastest I know of) implementation using Excel interop. 这是我(我所知道的最快)使用Excel interop的实现。 Although I looked carefully to release all (must have missed one), 2 Excel instances remain in the processes list, they are closed after the program ends.
尽管我仔细地考虑释放了所有(必须错过一个),但是2个Excel实例仍保留在进程列表中,但在程序结束后将它们关闭。
The key is to only have 2 Open Excel instances and to copy the data as a Block using Range.Value2
. 关键是只有2个Open Excel实例,并使用
Range.Value2
将数据复制为Block。
//Helper function to cleanup
public void ReleaseObject(object obj)
{
if (obj != null && Marshal.IsComObject(obj))
{
Marshal.ReleaseComObject(obj);
}
}
public void CopyIntoOne(List<string> pSourceFiles, string pDestinationFile)
{
var sourceExcelApp = new Microsoft.Office.Interop.Excel.Application();
var destinationExcelApp = new Microsoft.Office.Interop.Excel.Application();
// TODO: Check if it exists
destinationExcelApp.Workbooks.Open(pDestinationFile);
// for debug
//destinationExcelApp.Visible = true;
//sourceExcelApp.Visible = true;
int i = 0;
var sheets = destinationExcelApp.ActiveWorkbook.Sheets;
var lastsheet = destinationExcelApp.ActiveWorkbook.Sheets[sheets.Count];
ReleaseObject(sheets);
foreach (var srcFile in pSourceFiles)
{
sourceExcelApp.Workbooks.Open(srcFile);
// get extends
var lastRow = sourceExcelApp.ActiveSheet.Cells.Find("*", System.Reflection.Missing.Value,
System.Reflection.Missing.Value, System.Reflection.Missing.Value, XlSearchOrder.xlByRows,
XlSearchDirection.xlPrevious, false, System.Reflection.Missing.Value, System.Reflection.Missing.Value);
var lastCol = sourceExcelApp.ActiveSheet.Cells.Find("*", System.Reflection.Missing.Value, System.Reflection.Missing.Value,
System.Reflection.Missing.Value, XlSearchOrder.xlByColumns, XlSearchDirection.xlPrevious, false,
System.Reflection.Missing.Value, System.Reflection.Missing.Value);
var startCell = (Range) sourceExcelApp.ActiveWorkbook.ActiveSheet.Cells[1, 1];
var endCell = (Range) sourceExcelApp.ActiveWorkbook.ActiveSheet.Cells[lastRow.Row, lastCol.Column];
var myRange = sourceExcelApp.ActiveWorkbook.ActiveSheet.Range[startCell, endCell];
// copy the values
var value = myRange.Value2;
// create sheet in new Workbook at the end
Worksheet newSheet = destinationExcelApp.ActiveWorkbook.Sheets.Add(After: lastsheet);
ReleaseObject(lastsheet);
lastsheet = newSheet;
//its even faster when adding it at the front
//Worksheet newSheet = destinationExcelApp.ActiveWorkbook.Sheets.Add();
// change that to a good name
newSheet.Name = ++i + "";
var dstStartCell = (Range) destinationExcelApp.ActiveWorkbook.ActiveSheet.Cells[1, 1];
var dstEndCell = (Range) destinationExcelApp.ActiveWorkbook.ActiveSheet.Cells[lastRow.Row, lastCol.Column];
var dstRange = destinationExcelApp.ActiveWorkbook.ActiveSheet.Range[dstStartCell, dstEndCell];
// this is the actual paste
dstRange.Value2 = value;
//cleanup
ReleaseObject(startCell);
ReleaseObject(endCell);
ReleaseObject(myRange);
ReleaseObject(value);// cannot hurt, but not necessary since its a simple array
ReleaseObject(dstStartCell);
ReleaseObject(dstEndCell);
ReleaseObject(dstRange);
ReleaseObject(newSheet);
ReleaseObject(lastRow);
ReleaseObject(lastCol);
sourceExcelApp.ActiveWorkbook.Close(false);
}
ReleaseObject(lastsheet);
sourceExcelApp.Quit();
ReleaseObject(sourceExcelApp);
destinationExcelApp.ActiveWorkbook.Save();
destinationExcelApp.Quit();
ReleaseObject(destinationExcelApp);
destinationExcelApp = null;
sourceExcelApp = null;
}
I have tested it on small excel files and are curious how it behaves with larger files. 我已经在小型excel文件上对其进行了测试,并且很好奇它在较大文件中的行为。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.