Import two csv files to mysql database

Question

I need to import two csv files with different columns to mysql database table.

when I import the second file, table still with the data of the first csv file so I need a method or a test to update data if there is a difference and to add data that doesn't exist.

My code looks like this :

Method Import_Bilan for the first file and Import_data for the second one.

 private void Import_Bilan_Click(object sender, EventArgs e)
{
    DialogResult result = openFileDialog1.ShowDialog(); // Show the dialog.
    if (result == DialogResult.OK) // Test result.
    {
        string file = openFileDialog1.FileName;
        string[] f = file.Split('\\');

        // to get the only file name
        string fn = f[(f.Length) - 1];
        string path = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
        string dest = path + @"\upload\" + fn;

        //to copy the file to the destination folder
        File.Copy(file, dest, true);
        MessageBox.Show("File Uploaded !!");

        //to copy the file to the destination folder
        File.Copy(file, dest, true);

        MySqlConnection con = new MySqlConnection("datasource=localhost;database=***;port=3306;username=root;password=root;SslMode=none;AllowUserVariables=true");
        var msbl = new MySqlBulkLoader(con)
        {
            TableName = "**",
            FieldTerminator = ";",
            FileName = dest,
            NumberOfLinesToSkip = 1,

        };
        msbl.Columns.AddRange(new[] { "***", "***""@discard", "@discard","@discard", "@discard", "@discard"});

        msbl.Load();
        con.Close();

        MessageBox.Show("Data bind to database !!");
    }
}


private void Import_Data_Click(object sender, EventArgs e)
{
    DialogResult result = openFileDialog1.ShowDialog(); // Show the dialog.
    if (result == DialogResult.OK) // Test result.
    {
        string file = openFileDialog1.FileName;
        string[] f = file.Split('\\');

        // to get the only file name
        string fn = f[(f.Length) - 1];
        string path = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
        string dest = path + @"\upload\" + fn;

        //to copy the file to the destination folder
        File.Copy(file, dest, true);
        MessageBox.Show("File Uploaded !!");

        //to copy the file to the destination folder
        File.Copy(file, dest, true);

        MySqlConnection con = new MySqlConnection("datasource=localhost;database=***;port=3306;username=root;password=root;SslMode=none;AllowUserVariables=true");
        var msbl = new MySqlBulkLoader(con)
        {
            TableName = "****",
            FieldTerminator = ";",
            FileName = dest,
            NumberOfLinesToSkip = 1,

        };

        msbl.Columns.AddRange(new[] { "@discard", "***" });

        msbl.Load();
        con.Close();

        MessageBox.Show("Data bind to database !!");
    }
}

Answer 1

Based on the column names passed to msbl.Columns.AddRange , it appears that your two input CSV files have very different types of data and number of columns. (If this isn't correct, please edit your question with information about the structure of your CSV files.)

If this is true, I would suggest one of two different approaches:

Read CSV in C#

Use a library like CsvHelper to read the two CSV files in C# and join the data together (eg, by creating a Dictionary based on a shared key), then insert the rows (one at a time) into MySQL. This may end up being slower, and will probably take more code. So, instead you could…

Merge the tables in MySQL

Start with your code above, but load the CSV files into two temporary tables; let's call them exercices1 and exercices2 . (They can have the same schema as exercices , or you could create a custom schema for each CSV if you like.)

Then execute a MySQL INSERT INTO statement:

INSERT INTO exercices (all, the, output, columns, that, you, want)
SELECT e1.column1, e1.column2, e2.column1, e2.column2
FROM exercices e1 JOIN exercices e2
    WHERE e1.some_column = e2.some_column;

Obviously the exact details will depend on the exact structure of your CSV files, what data they contain, which columns they have in common (that you can join on), etc.

Once this is done, you can drop the two tables that you loaded the CSV data into.