Importing huge data from text file to sql using c# winforms

Question

I am wanting to add the ability to import massive CSV files into a table in the database. I've built an SSIS package that does this, but I just wanted to make sure that this was the correct way of doing it.

The text files are millions of rows with 50 columns. They don't open in notepad or notepad++ a lot of times. The SSIS package handles them with no problem and gets everything imported. Is SSIS the right way to deal with this? I just need to pass the file location parameter to the job and execute it right? Is there an easier way that I am overlooking?

Answer 1

The text files are millions of rows with 50 columns

Small. Why an SSIS package?

They don't open in notepad or notepad++

because both OPEN them - there is no need to open them and load them all into memory, a proper application can read them line by line.

Is SSIS the right way to deal with this?

No. SImply because the title is: Importing huge data from text file to sql using c# winforms

As Winforms can do that effectively - I am inserting around 100.000 rows into a database - in C# with quit eeasy coding (except some pages and a day to get sqlbulkcopy to work properly) and you say nothing about any transformations, SSIS just is another not needed technology AND makes things complicated (as in: more to install, or hav a packag on the server but then find a location for the file that the server can reach etc.

I am all for SSIS and if you have a larger SSIS infrastructure or do a lot of processing etc. and it makes architectural sense in a larger context - yes. But as the question stands, without additional reasons - absolutely not.

Heck, there is a good chance you can load the whole thing in one command into SQL server as SQL Server has some CSV processing capabilities:

http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/