SSIS - Fixed number of columns in flat file source

Question

I have a number of text files in a directory which have a set number of columns [6] separated by tabs. I read this into an SSIS package using a 'Flat File Source' block. If a file has more columns than the required number or if data is missing from any of the columns, I want to reject this file.

I have done some testing with various sample files. Whenever I add additional columns, the program accepts these files. It throws an error when there are less columns which is good.

But, is there a way of specifying that the file must have a certain amount of columns and that data must be present in each column?

I don't have much experience with SSIS so I would appreciate any suggestions.

Thanks

Answer 1

I would use a Script Task to do this.

You can use System.IO.StreamReader to open the file and read your header row, and then perform whatever validation you need on the resulting string.

I would also create a Boolean variable in the SSIS package, called something like 'FileIsValid', to which I would write (from the Script Task) True if the conditions are met, and False if they aren't. I would then use this to direct the package flow using precedence constraints.

Something like this:

public void Main()
{
    System.IO.StreamReader reader = null;

    try
    {
        Dts.Variables["User::FileIsValid"].Value = false;

        reader = new System.IO.StreamReader(Dts.Variables["User::Filepath"].Value.ToString());

        string header = reader.ReadLine();

        if (header.Trim() == "Column1\tColumn2\tColumn3\tColumn4\tColumn5\tColumn6")
            Dts.Variables["User::FileIsValid"].Value = true;

        reader.Close();
        reader.Dispose();

        Dts.TaskResult = (int)ScriptResults.Success;
    }
    catch
    {
        if (reader != null)
        {
            reader.Close();
            reader.Dispose();
        }

        throw;
    }
}

With regards to checking there is data in all columns, does this need to be for every row?

You could continue reading the lines with StreamReader and use regular expressions to check for something like this.

Answer 2

Expanding on Chris Mack:

If files do not have headers you can do a count.

char[] delim = new char[] {'\t'};
if(header.Split(delim).Length() == 5)
...