SSIS: How to read, omit by matching, and write records from pipe delimited flat file in C#

Question

I am a beginner in SSIS and C# programming, just got a new job and want to make a mark. I previously created a package to convert a tab delimited flat file to a SQL Server table, but the flat file was prepared manually for conversion. Now I need to automate the conversion of the pipe delimited file, which was created like a report and has several heading and sub-heading rows. The conversion from pipe to tab delimited is not a issue for me. However, I just can't find a way nor get any help online to understand how to read each record, determine it's content, and omit or write a record.

I put together the following SSIS C# script shown below coding from Main () only, but I am getting the following error, and I don't know why I am getting it. Can you help me out here?

ERROR - (Error at Script Task1: The binary code for the script is not found. Please open the script in the designer by clicking Edit Script button and make sure it builds successfully. Error at Script Task 1: There were errors during task validation.)

The script is supposed to:

1) Read each record in the pipe delimited flat file

2) check each line/record to determine if they contain the following values, and do not write records if they contain these values:

• Spaces

• Value - “Business Unit:” etc.

• Value - "Empl Id | Employee Name | Dept Id | Department | EE Home Phone | Emergency Contact Name | Primary | Telephone | Relationship"

• The last value is the heading. I want to write the 1st occurrence of this heading, but do not write any other occurrences of it afterwards.

SCRIPT:

public void Main()
{
  string SourcePath = Dts.Variables["User::Source_Path"].Value.ToString();
  string DestinationPath = Dts.Variables["User::Destination_Path"].Value.ToString();
  string Line = string.Empty;

  try
  {
    using (StreamReader sr = new StreamReader(SourcePath))
    using (StreamWriter ds = new StreamWriter(DestinationPath))
    {
      while ((Line = sr.ReadLine()) != null)
      { 
        // MessageBox.Show(Line);
        if (Line == " ")
          Console.WriteLine("Blank Line");
        else
          if (Line == "Business Unit: 069 - DEPT OF XXXX XXXXX")
            Console.WriteLine("069 Heading");
          else
            if (Line == "Business Unit: 071 - DEPT. OF YYYYY YYYYYYY")
              Console.WriteLine("071 Heading");
            else
              if (Line == "Empl Id | Employee Name | Dept Id | Department | EE Home Phone | Emergency Contact Name | Primary | Telephone | Relationship")
                Console.WriteLine("Main Heading");

        // write to destination file
        ds.WriteLine(Dts.Variables["User::Destination_Path"].Value);
      }
      // close the stream
      ds.Close();
      //string data = sr.ReadToEnd();
      //MessageBox.Show(data);
    }
    Dts.TaskResult = (int)ScriptResults.Success;
  }
  catch (Exception ex)
  {
    MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK);
  }

Answer 1

The reason the file path is being printed is because you're calling StreamWriter.WriteLine() using the variable that holds the file path, as opposed using the text itself. The file path will be used when initializing the StreamWriter and the text when the WriteLine() method is called. I'd also recommend storing the text you don't want to write in string variables as done below. As necessary, you can add additional strings that will be filtered out (ie the "TextToOmit" strings below). You may need to may a couple modifications for the exact strings you want to omit, but the code below will keep on the first header and remove the subsequent strings. You can remove the Close() method as it isn't necessary since this will be closed upon exiting the using block. Lines with white-space are also filtered out by the String.IndexOf method, which returns -1 when the text is not found. You mentioned concerns about getting a direct match, you can use the IndexOf method with the StringComparision.CurrentCultureIgnoreCase parameter to match the first occurrence without case-sensitivity. However with this approach you will want to be sure that the text you're omitting doesn't occur in any records that need to be kept, an example of this is below in initial code snippet. I'm assuming that you want to write each record on a new line, which is why the Environment.NewLine property is added when building the output text.

string sourcePath = Dts.Variables["User::Source_Path"].Value.ToString();
string destinationPath = Dts.Variables["User::Destination_Path"].Value.ToString();
string line = string.Empty;
string outputText = string.Empty;
string headerText = "YourHeaderLine";
string secondTextToOmit = "TextThatNeedsToBeOmitted";
string thirdTextToOmit = "TextThatNeedsToBeOmitted";
int headerCount = 0;
try
{
    using (StreamReader sr = new StreamReader(sourcePath))
    {
        while ((line = sr.ReadLine()) != null)
        {
            //only write first occurance
            if (line == headerText && headerCount == 0)
            {
                outputText = outputText + line + Environment.NewLine;
                headerCount++;
            }
            else
             //store text in variables to do checks all in same if statement
             //IndexOf looks for while space
             if (line.IndexOf(' ') < 0 && line != headerText
                && line != secondTextToOmit && line != thirdTextToOmit)
            {
                outputText = outputText + line + Environment.NewLine;
            }
            //initialize StreamWriter using file path
            using (StreamWriter writer = new StreamWriter(destinationPath))
            {
                //write the string using filtered text
                writer.WriteLine(outputText);
            }
        }
    }
}

Match Without Case-Sensitivity Example:

line.IndexOf(thirdTextToOmit, 0, StringComparison.CurrentCultureIgnoreCase) < 0