提高将大型execl文件转换为对象列表的性能

Question

I know this question have been asked multiple times . But I could not find much help from anyone of those.

I don't want to convert the excel into data table but I want it to be converted to a list of objects and sent to server side for processing.

If it has more than 2K rows it should throw an error. Currently what I am doing is something like :

   using (var excel = new ExcelPackage(hpf.InputStream))
    {
        var ws = excel.Workbook.Worksheets["Sheet1"];

        for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
        {
            if (ws.Cells[rw, 1].Value != null)
            {
                int headerRow = 2;

                GroupMembershipUploadInput gm = new GroupMembershipUploadInput();

                for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++)
                {
                    var s = ws.Cells[rw, col].Value;

                    if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id"))
                    {
                        gm.cnst_mstr_id = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
                    }
                    else if (ws.Cells[headerRow, col].Value.ToString().Equals("Prefix of the constituent(Mr, Mrs etc)"))
                    {
                        gm.cnst_prefix_nm = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
                    }
                    else if (ws.Cells[headerRow, col].Value.ToString().Equals("First Name of the constituent(Mike)"))
                    {
                        gm.cnst_first_nm = (ws.Cells[rw, col].Value ?? (Object)"").ToString();
                    }
                    .....................
                    .....................


                    }
            }

                    iUploadedCnt = iUploadedCnt + 1; //Increase the count by 1
                }

                if (lgl.GroupMembershipUploadInputList.Count < 2003) //Check for the uploaded list count
                {

                       //throw the error

                 }

But this approach seems slow.

Conversion of the excel to list seems slow to me. For example , when I upload more than 2k records , the list gets converted first to list and then the count is checked if more than 2003 . This process is definitely slower.

How can it be achieved in a faster /better way ?

Answer 1

You do a lot of repeated string processing which is unnecessary. For each row you check the column headers again if they fit some predefined value. (for instance if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id")) .

You could do this once before you start parsing all rows and create for instance a Dictionary<int, SomeEnum> which maps the column number to a specific enum value. When parsing the rows you then can make a quick lookup in the dictionary, which column maps to which property.

Furthermore, you define a var s = ws.Cells[rw, col].Value; but never use it. Instead, you read this cell value again, when you assign it to a property of your object. You could just make the necessary conversions and checks here, and then use only s ;

// define this enum somewhere
enum ColumPropEnum {
   cnst_mstr_id,  cnst_prefix_nm, ...
}

//define this prop somewhere
Dictionary<int, ColumnPropEnum> colprops = new Dictionary<int, ColumnPropEnum>();

//do this once before processing all rows
for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++) {
    if (ws.Cells[headerRow, col].Value.ToString().Equals("Existing Constituent Master Id")) 
        colprops.Add(col, ColumnPropEnum.cnst_mstr_id);
    else if (ws.Cells[headerRow, col].Value.ToString().Equals(" ..."))
        colprops.Add(col, ColumnPropEnum.cnst_prefix_nm);
    ...
}


//now use this dictionary in each row    
for (int rw = 4; rw <= ws.Dimension.End.Row; rw++)
{
....
    for (int col = ws.Dimension.Start.Column; col <= ws.Dimension.End.Column; col++) {

        //the single ? checks, whether the Value is null, if yes it returns null, otherwise it returns ToString(). Then the double ?? checks whether the result if the operation is null, if yes, it assigns "" to s, otherwise the result of ToString(); 
        var s = ws.Cells[rw, col].Value?.ToString() ?? "";
        ColumnPropEnum cp;
        if (colpros.TryGetValue(col, out cp)) {
            switch (cp) {
                case cnst_mstr_id: gm.cnst_mstr_id = s; break;
                case cnst_prefix_nm: gm.cnst_prefix_nm = s; break;
                ...
            }
        }
    }

}

I'm not sure at which position you add this object to a list or upload it to the server, as this is not part of the code. But it could be faster, to first check only the first column of each row if you have the necessary count of non-null values and throw an error if not and do all the other processing only if you didn't throw the error.

int rowcount = 0;
//If you need at minimum 2000 rows, you can stop after you count 2000 valid rows
for (int rw = 4; rw <= ws.Dimension.End.Row && rowcount < 2000; rw++)        
{
    if (ws.Cells[rw, 1].Value != null) rowcount++
}

if (rowcount < 2000) {
    //throw error and return
} 

//else do the list building and uploading