insert the csv file data into clickhouse database in the console application using c#

Question

I need to insert the csv file data into clickhouse database in the console application using c#. while doing in sql database the data is adding to table and in the clickhouse it is not adding data into the table.

Data is unable to add in clickhouse database and also i am not getting any errors

    static void Main()
    {
        string csv_file_path = @"C:\Users\thummala.naveen\Downloads\Employee.csv";
       
      InsertDataIntoSQLServerUsingSQLBulkCopy();

    }
    
    public static void InsertDataIntoSQLServerUsingSQLBulkCopy()
    {

        
        SqlConnection con = new SqlConnection(@"data source=10.1.230.49\standard2019;initial catalog=thummala.naveen@hcl.com;uid=sa;password=dbnms#123;integrated security=false;MultipleActiveResultSets=True;App=EntityFramework");
        
        string filepath = "C:\\Users\\thummala.naveen\\Downloads\\Employee.csv";
        StreamReader sr = new StreamReader(filepath);
        string line = sr.ReadLine();
        string[] value = line.Split(',');
        DataTable dt = new DataTable();
        DataRow row;
        foreach (string dc in value)
        {
            dt.Columns.Add(new DataColumn(dc));
        }

        while (!sr.EndOfStream)
        {
            value = sr.ReadLine().Split(',');
            if (value.Length == dt.Columns.Count)
            {
                row = dt.NewRow();
                row.ItemArray = value;
                dt.Rows.Add(row);
            }
        }
        
        ClickHouse.Client.ADO.ClickHouseConnection conn = new ClickHouse.Client.ADO.ClickHouseConnection(@"Compress=True;CheckCompressedHash=False;Compressor=lz4;Host=10.1.162.59;Port=8123;User=default;Password=clickhouse@123;SocketTimeout=600000;Database=naveentest;");
        ClickHouseBulkCopy bcs = new ClickHouseBulkCopy(conn.ConnectionString);
        bcs.DestinationTableName = "Emp";
        bcs.BatchSize = dt.Rows.Count;
        using var csvs = CsvDataReader.Create("C:\\Users\\thummala.naveen\\Downloads\\Employee.csv");

        bcs.WriteToServerAsync(csvs);
        conn.Close();
    }



}

}

Answer 1

While this is technically NOT an answer, this should help you find the problem yourself: This rewrite of your code, would be infinitely easier to test, in a unit test framework. May I suggest xunit? But any unit test framework will do.

The structure isn't perfect, but should lead you down the right path of what you need to do, to make your code more testable. (And therefore easier to debug)

With this structure, you can "Mock" (ie. moq framework) your individual behaviours out, and write coverage tests, and whitebox tests, and precisely where your problem is.

For you main method class, default name program, I suppose this will do:

using System.Data;

namespace StackDemoConsoleApp
{
    public class Program
    {
        private static readonly IRowCountHandler _rowCountHandler = new RowCountHandler(new EmployeeFileHandler(), new DataTablePopulator(), new DataTablePreparer());
        private static readonly IPersistenceWriter _persistenceWriter = new PersistenceWriterClickHouse();

        public static void Main(string[] args)
        {
            DataTable dataTableForRowCount = _rowCountHandler.FindRowCount();
            _persistenceWriter.WriteToPersistence(dataTableForRowCount);
        }
    }
}

The rest of these are either interfaces or classes, that you should have individually named files for, in your project explorer:

    using System.Data;
    using System.IO;
    
    namespace StackDemoConsoleApp
    {
        public class DataTablePopulator : IDataTablePopulator
        {
            public DataTable PopulateDataTable(StreamReader streamReader, ref string[] value, DataTable dataTableForRowCount)
            {
                while (!streamReader.EndOfStream)
                {
                    value = streamReader.ReadLine().Split(',');
                    if (value.Length == dataTableForRowCount.Columns.Count)
                    {
                        var row = dataTableForRowCount.NewRow();
                        row.ItemArray = value;
                        dataTableForRowCount.Rows.Add(row);
                    }
                }
    
                return dataTableForRowCount;
            }
        }
    }


    using System.Data;
    
    namespace StackDemoConsoleApp
    {
        public class DataTablePreparer : IDataTablePreparer
        {
            public DataTable PrepareDataTableStructure(string[] value)
            {
                DataTable dataTableForRowCount = new DataTable();
                foreach (string dc in value)
                {
                    dataTableForRowCount.Columns.Add(new DataColumn(dc));
                }
    
                return dataTableForRowCount;
            }
        }
    }


    using System.IO;
    
    namespace StackDemoConsoleApp
    {
        public class EmployeeFileHandler : IEmployeeFileHandler
        {
            public void ReadEmployeeFile(out StreamReader sr, out string[] value)
            {
                string filepath = "C:\\Users\\thummala.naveen\\Downloads\\Employee.csv";
                sr = new StreamReader(filepath);
                string line = sr.ReadLine();
                value = line.Split(',');
            }
        }
    }

    using System.Data;
    using System.IO;
    
    namespace StackDemoConsoleApp
    {
        public interface IDataTablePopulator
        {
            DataTable PopulateDataTable(StreamReader sr, ref string[] value, DataTable dataTableForRowCount)
        }
    }


using System.Data;
using System.IO;

namespace StackDemoConsoleApp
{
    public interface IDataTablePopulator
    {
        DataTable PopulateDataTable(StreamReader sr, ref string[] value, DataTable dataTableForRowCount)
    }
}

using System.IO;

namespace StackDemoConsoleApp
{
    public interface IEmployeeFileHandler
    {
        void ReadEmployeeFile(out StreamReader sr, out string[] value);
    }
}

using System.Data;

namespace StackDemoConsoleApp
{
    public interface IPersistenceWriter
    {
        void WriteToPersistence(DataTable dataTableForRowCount);
    }
}

using System.Data;

namespace StackDemoConsoleApp
{
    public interface IRowCountHandler
    {
        DataTable FindRowCount();
    }
}

using System.Data;

namespace StackDemoConsoleApp
{
    public class PersistenceWriterClickHouse : IPersistenceWriter
    {
        public void WriteToPersistence(DataTable dataTableForRowCount)
        {
            ClickHouse.Client.ADO.ClickHouseConnection conn = new ClickHouse.Client.ADO.ClickHouseConnection(@"REDACTED");
            ClickHouseBulkCopy bcs = new ClickHouseBulkCopy(conn.ConnectionString);
            bcs.DestinationTableName = "Emp";
            bcs.BatchSize = dataTableForRowCount.Rows.Count;
            using var csvs = CsvDataReader.Create("C:\\Users\\thummala.naveen\\Downloads\\Employee.csv");
            bcs.WriteToServerAsync(csvs);
            conn.Close();
        }
    }
}

using System.Data;
using System.IO;

namespace StackDemoConsoleApp
{
    public class RowCountHandler : IRowCountHandler
    {
        private readonly IEmployeeFileHandler _employeeFileHandler;
        private readonly IDataTablePreparer _dataTablePreparer;
        private readonly IDataTablePopulator _dataTablePopulator;

        public RowCountHandler(IEmployeeFileHandler employeeFileHandler, IDataTablePopulator dataTablePopulator, IDataTablePreparer dataTablePreparer)
        {
            _employeeFileHandler = employeeFileHandler;
            _dataTablePopulator = dataTablePopulator;
            _dataTablePreparer = dataTablePreparer;
        }

        public DataTable FindRowCount()
        {
            StreamReader sr;
            string[] value;
            _employeeFileHandler.ReadEmployeeFile(out sr, out value);
            DataTable dataTableForRowCount= _dataTablePreparer.PrepareDataTableStructure(value);
            return _dataTablePopulator.PopulateDataTable(sr, ref value, dataTableForRowCount);
        }
    }
}

In this structure, fx. It was easy to see that: SqlConnection con = new SqlConnection(@"redacted");

Does nothing.

The code is now somewhat easier to read and understand.

And since the code is now split up into much smaller methods, it is much easier for you to ask for help, because the code base you are presenting to anyone else, is much smaller, and much more focused on the actual problem.

There are actually loads more benefits to the code written this way, than just this. But This post is already too long.

Just to be clear, there is a ton of renaming I left out, because, seriously, I can't be bother to correct it all. There are just too many "don't do" in this code...

But please make your code testable via DI, SOLID and Clean Code.

This will make your code quality way better, enable others to help you quicker, help yourself, understand and debug your own code, and make your code testable.