在 SSIS 中处理具有多种格式和不同记录集的平面文件

Question

I am trying to process a Pipped separated flat file using SSIS. The file has two different types of records. each record set has has different it's own header row and a trailer row. They need to go to two different table.

The trailer row has the row count for each record set while the header row contains column names for the records. So in essence it's like two types of files but in one file.

I've tried several solutions including using a conditional split but I haven't been able to achieve this. I know you could do this using a script component and C# but I haven't been able to achieve this. I've attached an image to show file format.

This is what I have tried so far.

1. I edited the flat file connection to ragged right so that the output comes in one column.
2. I then created a script component as source. The idea was to read the file line by line using stream reader, then create 4 output buffers. 2 for column headers and 2 for the different detail rows then set the script to stop when it reaches the trailer row. My intention was to merge each header row the respective detail rows then save them to the relevant tables
3. I then used C# code which I got from my research. I picked this up from the Microsoft site.

I used the Code Below:

public class ScriptMain : UserComponent
{
    private StreamReader textReader;
    private string RTWFile;

    public override void AcquireConnections(object Transaction)
    {
        IDTSConnectionManager100 connMgr = this.Connections.RTWCon;
        RTWFile = (string)connMgr.AcquireConnection(null);
    }

    public override void PreExecute()
    {
        base.PreExecute();
        textReader = new StreamReader(RTWFile);
    }

    public override void CreateNewOutputRows()
    {
        string nextLine;
        string[] columns;

        char[] delimiters;
        delimiters = "|".ToCharArray();

        nextLine = textReader.ReadLine();
        while (nextLine != null)
        {
            columns = nextLine.Split(delimiters);
            {
                HeadersBuffer.AddRow();
                HeadersBuffer.EmployeeNumber = columns[0];
                HeadersBuffer.LegacyStaffID = columns[1];
                HeadersBuffer.FirstName = columns[2];
                HeadersBuffer.LastName = columns[3];
                HeadersBuffer.PassportIssuingCountry = columns[4];
                HeadersBuffer.PassportType = columns[5];
                HeadersBuffer.PassportNumber = columns[6];
                HeadersBuffer.PassportIssuingAuthority =columns[7];           
                HeadersBuffer.PassportIssueDate = columns[8];
                HeadersBuffer.PassportExpirationDate = columns[9];
            }
            nextLine = textReader.ReadLine();
        }
    }

    public override void PostExecute()
    {
        base.PostExecute();
        textReader.Close();
    }
}

Image of the flat file format:

OutPutBuffers:

FlatFileConfiguration:

Answer 1

Do you need to extract the information from Trailer rows also?

Separating the file into two files is the cleanest way to do this. We can do this with a script task once we understand your requirements.

UPDATE:

Add Script Task and Provide FilePath as Read Variable

Edit Script task and add these to namespaces region at the top

using System.IO;
using System.Text;
using System.Collections.Generic;

public void Main()
{
    try
    {
        String InputFilePath = Dts.Variables["User::FilePath"].Value.ToString();
        string InputFolder = Path.GetDirectoryName(InputFilePath);
        string TrailerLine = "TotalRow";
        bool FirstFile = true;
        string line;
        List<string> FirstFileLines, SecondFileLines;

        // Read the file and display it line by line.  
        System.IO.StreamReader file =
            new System.IO.StreamReader(InputFilePath);
        FirstFileLines = new List<string>();
        SecondFileLines = new List<string>();
        while ((line = file.ReadLine()) != null)
        {
            if (line.Contains(TrailerLine))
            {
                FirstFile = false;
                continue;
            }

            if (FirstFile) FirstFileLines.Add(line);
            else SecondFileLines.Add(line);
        }

        File.WriteAllLines(InputFolder + @"\FirstFile.txt", FirstFileLines.ToArray());
        File.WriteAllLines(InputFolder + @"\SecondFile.txt", SecondFileLines.ToArray());
        file.Close();
        Dts.TaskResult = (int)ScriptResults.Success;
    }
    catch (System.Exception ex)
    {
        MessageBox.Show(ex.Message.ToString());
        Dts.TaskResult = (int)ScriptResults.Failure;
    }

}