Read text file from text line and insert into SQL Server database using C#

Question

I am trying to read in a text file using C# and insert the data from that text file (skipping headers) into a SQL Server table.

I was using bulk insert, but have since learned that using bulk insert to skip headers in a text file is misusing the bulkcommand

SET @bulkCommand = ' BULK INSERT ' + @databaseName + ' .dbo.ImportTable
FROM ''' + @fileName + '''
WITH (ROWTERMINATOR = ''\n'',
      FIELDTERMINATOR = ''\t'',
      DATAFILETYPE = ''char'',
      MAXERRORS = 0,
      FIRSTROW = ' + CAST (@Data2 AS VARCHAR(10)) + ')';

I have already created a table using the column headers (Column 1-13).

There is nothing within the text file that I am able to order by during the bulk insert so I find that the bulk insert reads in the previous headers as a single row and misses some of the data I am trying to import.

I think that the best approach is to read the text file using C# and insert the 'DATA' directly into my SQL Server table. So when the C# script reaches line='DATA' + 1 it imports it to my SQL Server table.

The text file looks something like this;

[Header]
Blah1   1.9.4
Blah2   8/13/2019 9:08 AM
Blah3   Blah_nnnn_C.bpm
Blah4   14376
Blah5   18004
Blah6   116
Blah7   672
[Data]
Column1 Column2 Column3 Column4 Column5 Column6 Column7 Column8 Column9 Column10  Column11  Column12  Column13      
ARS-BFGL-BAC-10972  UK160457403929  G   C   A   B   C   G   0.8651  0.677   0.718   0.4864  -0.2590
ARS-BFGL-BAC-10975  UK160457403929  A   G   A   B   A   G   0.9303  0.604   0.668   0.5116  0.1358
ARS-BFGL-BAC-11025  UK160457403929  G   G   B   B   C   C   0.9092  0.021   0.985   1.0000  -0.0762
ARS-BFGL-BAC-11044  UK160457403929  T   T   A   A   A   A   0.9626  0.941   0.003   0.0000  -0.0977
ARS-BFGL-BAC-11193  UK160457403929  A   A   A   A   A   A   0.9544  0.538   0.000   0.0000  -0.0775
ARS-BFGL-BAC-11215  UK160457403929  A   G   A   B   A   G   0.8400  0.146   0.296   0.5626  -0.7361
ARS-BFGL-BAC-11218  UK160457403929  A   A   A   A   A   A   0.9496  1.186   0.020   0.0000  0.0131
ARS-BFGL-BAC-11276  UK160457403929  A   G   A   B   A   G   0.9416  0.577   0.570   0.5320  -0.1528
ARS-BFGL-BAC-11723  UK160457403929  T   C   A   B   A   G   0.9610  0.566   0.627   0.5039  -0.0593
ARS-BFGL-BAC-11724  UK160457403929  T   C   A   B   A   G   0.8602  0.296   0.600   0.5375  0.1308
ARS-BFGL-BAC-11748  UK160457403929  C   G   A   B   C   G   0.8581  0.506   0.544   0.4631  -0.0229
ARS-BFGL-BAC-11750  UK160457403929  G   G   B   B   G   G   0.9336  0.010   0.783   1.0000  0.0400
ARS-BFGL-BAC-11783  UK160457403929  A   G   A   B   A   G   0.9701  0.266   0.316   0.4858  -0.0636
ARS-BFGL-BAC-1180   UK160457403929  T   C   A   B   A   G   0.9636  0.304   0.369   0.5165  -0.2222
ARS-BFGL-BAC-11805  UK160457403929  G   G   B   B   G   G   0.7832  0.062   1.414   1.0000  0.0778
ARS-BFGL-BAC-11867  UK160457403929  T   T   A   A   A   A   0.9115  1.220   0.054   0.0000  -0.0038
ARS-BFGL-BAC-11913  UK160457403929  G   G   B   B   C   C   0.8620  0.033   1.201   1.0000  -0.0393
ARS-BFGL-BAC-12159  UK160457403929  T   C   A   B   A   G   0.9625  0.434   0.468   0.4957  0.0169
ARS-BFGL-BAC-12297  UK160457403929  A   G   A   B   A   G   0.9440  0.503   0.536   0.4837  -0.0246
ARS-BFGL-BAC-12468  UK160457403929  T   C   A   B   A   G   0.9618  0.578   0.621   0.5011  -0.0039
ARS-BFGL-BAC-12579  UK160457403929  A   A   A   A   A   A   0.8726  0.965   0.055   0.0000  0.1988
ARS-BFGL-BAC-12893  UK160457403929  C   C   B   B   G   G   0.9577  0.015   1.021   1.0000  -0.0088
ARS-BFGL-BAC-1302   UK160457403929  T   C   A   B   A   G   0.9423  0.747   0.648   0.4974  0.1031
ARS-BFGL-BAC-13210  UK160457403929  C   C   B   B   G   G   0.8578  0.019   1.535   1.0000  -0.0282
ARS-BFGL-BAC-13545  UK160457403929  A   A   A   A   A   A   0.9562  0.647   0.013   0.0000  -0.5184
ARS-BFGL-BAC-13568  UK160457403929  T   T   A   A   A   A   0.9588  0.784   0.007   0.0000  0.0387

I can simply write out to another text file, say selecting the top 30 lines, but don't know how to do this from line 'DATA' + 1 and insert into my sql table?

using System;
using System.IO;
using System.Linq;

namespace Generic
{
    class ReadFromFile
    {
        static void Main()
        {
            if(File.Exists(@"outputfile.txt"))
            {
                File.Delete(@"outputfile.txt");
            }

            string[] lines = System.IO.File.ReadLines(@"inputfile.txt").Take (30). ToArray();

            foreach (string line in lines)
            {
                Console.WriteLine(line);

                FileStream fs = new FileStream("outputfile.txt",     FileMode.Append);
                TextWriter tmp = Console.Out;
                StreamWriter sw = new StreamWriter(fs);

                Console.SetOut(sw);
                Console.WriteLine(line);

                Console.SetOut(tmp);
                sw.Close();
            }
            //Console.ReadKey();
        }
    }
}

I don't have much experience with C# and don't know where to begin really. So would be great if someone could help me out.

Answer 1

void Main()
{
  using (var stream = File.OpenText(@"d:\temp\SampleTextFile.txt"))
  {
    while (stream.ReadLine() != "[Data]") { };
    stream.ReadLine();


    while (!stream.EndOfStream)
    {
      string line = stream.ReadLine();
      // parse the line and do whatever. ie:
      // line.Split(' ').Where(l => !string.IsNullOrEmpty(l));
    }
  }
}

If the real text file is as simple as the one you gave as a sample, then you could simply use Split and get 13 items per row. The text file looks like a Fixed Width Data aka System Data Format (SDF), which you can parse using Substring().

PS: Field terminator \\t ? Are you sure that this file is tab delimited?