从整数和小数字符串中删除字符的最有效方法

Question

I'm processing raw US Census data into a SQL Server database. The tar file when unzipped yields a little over 14,000 CSV files needing to be processed into 266 different database tables. I have to loop over each CSV file and append a header to the file so SSIS can ETL the raw data into a targeted SQL Server table.

Each CSV file's first 6 columns are exactly the same. The remaining columns per file are different. The data in the remaining columns are mostly numeric values (integers and decimals). However, the Census Bureau adds characters called 'jam' values representing why there is no value. I need to replace these jam values with null or an empty string because the target database table columns are DECIMALS and jam values cause SSIS to fail insertion.

So, I have a C# (DotNet Core) class library looping over 14K files. For each file I have to do the following:

1. create a StringBuilder variable
2. append row header to StringBuilder so SSIS works
3. loop over each row in file
4. for each row, I have to split out the first 6 columns because I need those strings in target table. I then split out remaining columns because I have to remove jam values leaving numeric data
5. combine first 6 columns and cleaned data back into a row
6. append newly cleaned row to StringBuilder
7. after finishing looping through all rows, write StringBuilder to destination folder where SSIS will load into database.

I have 3 nested loops:

1. loop over 14000 files
2. for each file, loop over each row
3. for each row, loop over columns removing characters

Here's my code for looping over each file:

    private static Boolean BuildCensusDataFileWithHeader(String censusDataFilePath, String rowHeader, String censusDataDestinationFilePath)
    {
        try
        {
            // BUILD NEW FILE WITH HEADER
            StringBuilder currentContent = new StringBuilder();
            currentContent.Append(rowHeader + Environment.NewLine);

            //RETRIEVE ALL LINES IN TARGET FILE
            List<String> rawList = File.ReadAllLines(censusDataFilePath).ToList();

            // LOOP THROUGH EACH LINE AND REMOVE ANY STRINGS IN COLUMNS AFTER COLUMN 6
            // NOTE: COLUMNS 1-6 CONTAINS STRINGS NEEDED IN DATABASE
            foreach (var row in rawList)
            {
                //TURN COMMA DELIMITED ROW OF DATA INTO ARRAY
                String[] rowArray = row.Split(",");

                // PEEL OFF FIRST 6 COLUMNS TO BE KEPT AS IS
                IList<String> goodStrings = rowArray.Take(6).ToList();

                // RETRIEVE REMAINING COLUMNS TO BE CLEANED OF STRINGS
                IList<String> stringsToNullList = rowArray.Skip(6).ToList();

                // REMOVE ALL STRINGS
                stringsToNullList.OnlyDecimalValues();

                // PUT GOOD COLUMNS AND CLEANED COLUMNS BACK TOGETHER AS A ROW
                var cleanedRow = $"{String.Join(",", goodStrings)},{String.Join(",", stringsToNullList)}";

                // APPEND ROW TO NEW DOCUMENT TO BE WRITTEN TO TARGET DIRECTORRY CONTAINING CLEANED DATA
                currentContent.Append(cleanedRow + Environment.NewLine);
            }

            File.WriteAllText(censusDataDestinationFilePath, currentContent.ToString());

            return true;
        }
        catch (Exception ee)
        {
            string temp = ee.Message;
            return false;
        }
    }

Here's my extension methods replacing characters with empty space:

    public static void OnlyDecimalValues(this IList<String> stringToClean)
    {
        for (int i = 0; i < stringToClean.Count; ++i)
        {
            stringToClean[i] = (stringToClean[i].IsDecimal()) ? stringToClean[i] : "";
        }
    }

    public static bool IsDecimal(this string text)
    {
        decimal test;
        return decimal.TryParse(text, out test);
    }

This is all working through brute force programming. Is there a more efficient way to do this?

Thank you for your time.

Answer 1

I have two suggestions to speed it up. First, since you don't do anything with a resulting parsed decimal value, you can use a regular expression to check if string contains numbers only. It is faster than using a TryParse. I used a Stopwatch to check the speed and this way yields a slightly better performance for "false" cases and significantly better performance for "true" cases. So, IsDecimal method would become:

private static bool IsDecimal(string text)
{
    var regex = @"^-?(0|[1-9]\d*)(\.\d+)?$";
    return Regex.Match(text, regex).Success;
}

Second suggestion, is to transform an if-else block into just an if block. So, this line:

stringToClean[i] = (stringToClean[i].IsDecimal()) ? stringToClean[i] : "";

would become this:

if (!stringToClean[i].IsDecimal())
{
    stringToClean[i] = "";
}

Answer 2

I recommend revisiting process design. Use the power of sql and ssis in the right balance. Use ssis to loop through all files in the folder and to load raw text rows into newly created raw table. Then use sql code to do the rest of the processing. You can use charindex or patIndex functions to split the raw rows and one benifit of SQL would be a massive reduction in runtime because you will be processing the entire batch in a single transaction for a given file.

Another likely upside would be that you may just have to create one raw table for all the different files, with three columns - id, fileName , rawText. So the design would look something like:

Steps performed in SSIS

• Create a StringBuilder variable. Append row header to StringBuilder so SSIS works loop over each row in file.

Steps performed in SQL

• Split out the first 6 columns to get strings in target table & split out remaining columns to remove jam values leaving numeric data using a single select statement using patindex or charindex functions combined with replace function for nullifying jam values.