Bulk Data extraction to DB File in C#

Question

I have Huge data say 45,00,000 rows of data in notepad file ,
I split that gaint file into small files,
I have the data as follows:

('1','dsamp','tty','tmp'....)
and so on

Now i am reading the files one by one and using the insert script and a piece of C# code I am writing them to a .mdf file , but when i get some error I am unable to find where the error is and i want to start from beginning and insert from row 0.
Is there any best way or code or tool to do this My code looks like this

    private void Form1_Load(object sender, EventArgs e)
    {
        int i = 0;
        try
        {
            string const_state = "INSERT INTO Authors1 VALUES";
            string conn = @"Data Source=(LocalDB)\v11.0;AttachDbFilename=c:\users\srikanth\documents\visual studio 2013\Projects\WindowsFormsApplication1\WindowsFormsApplication1\SampleDB.mdf;Integrated Security=True;Connect Timeout=30";
            SqlConnection cn = new SqlConnection(conn);
            cn.Open();
            SqlCommand cmd = new SqlCommand();
            cmd.Connection = cn;
            string line;

            System.IO.StreamReader file = new System.IO.StreamReader("C:\\Users\\Public\\New1.txt");
            while ((line = file.ReadLine()) != null)
            {
                line = line.Trim();
                line = line.TrimEnd(',', ',',',', '.');
                cmd.CommandText = const_state + line+";";
                cmd.ExecuteNonQuery();
                i++;
            }
            MessageBox.Show(i.ToString());

            file.Close();
        }
        catch(Exception ex)
        {
            MessageBox.Show(i.ToString());
            MessageBox.Show(ex.ToString());
        }

    }
}

}

Thanks in Advance

Answer 1

What I would do is having a try/catch block for your ExecuteNonQuery() call. Something like this:

        while ((line = file.ReadLine()) != null)
        {
            line = line.Trim();
            line = line.TrimEnd(',', ',',',', '.');
            cmd.CommandText = const_state + line+";";
            try 
            {
                cmd.ExecuteNonQuery();
            }
            catch
            {
                // dump cmd.CommandText somewhere as well as 
                // the actual exception details
                //
                // that'll give you two things: 1) the specific 
                // issue, and 2) the actual INSERT statement that 
                // failed
            }
            i++;
        }

See my comments in the catch { } block on how I would handle an INSERT error

By having the try/catch around the ExecuteNonQuery() call, you'll have the granular issue on which INSERT statement failed, as well as the particular exception with the error. The other benefit to this is it would allow you to continue execution instead of bubbling that exception up to the outer try/catch logic. Unless, of course, you want to stop execution, in which case you can just rethrow the exception from the inner catch { } block. It all depends on how you want a failure handled.

Note: for your outer try/catch block, you should include a finally { } where you call SqlConnection.Dispose() to release the connection and dispose of the object ( cn.Dispose() ).