How do I load text files greater than the 64 kb buffersize limit?

Question

I'm trying to load text files (.aspx, .cs, html, etc) into a sql server 2008 database. I'm able to load all files that are less than 64 kb so far. I have two questions; How do I get around the 64 kb limit, and is the method I'm using the best way to do this?

Thanks for the help.

Database:

file_length int,
file_path varchar(250),
file_string varchar(MAX)

Code:

private static void Load_Files()
{
    string source = HttpContext.Current.Server.MapPath("~/website/");

    DirectoryInfo di = new DirectoryInfo(source);
    FileInfo[] files = di.GetFiles();

    foreach (FileInfo f in files)
    {
        string sourceFile = f.FullName;

        FileStream fs_reader = new FileStream(sourceFile, FileMode.Open, FileAccess.Read);
        StreamReader reader = new StreamReader(fs_reader);
        string content = reader.ReadToEnd();

        Int32 file_length = content.Length;

        string CS = ConfigurationManager.ConnectionStrings["SQL_CS"].ConnectionString;
        SqlConnection SQL_Conn_01 = new SqlConnection(CS);

        string SQL_01 = "INSERT INTO Page_File_Store (file_length, file_path, file_string) VALUES (@file_length, @file_path, @file_string)";
        SqlCommand SQL_File_Load = new SqlCommand(SQL_01, SQL_Conn_01);
        SQL_File_Load.Parameters.Add(new SqlParameter("@file_length", file_length));
        SQL_File_Load.Parameters.Add(new SqlParameter("@file_path", sourceFile));
        SQL_File_Load.Parameters.Add(new SqlParameter("@file_string", content));

        SQL_Conn_01.Open();
        SQL_File_Load.ExecuteNonQuery();
        SQL_Conn_01.Close();

        reader.Close();
    }
}

Answer 1

In SQL server 2008, Microsoft added a new data type, called FileStream. It "integrates the SQL Server Database Engine with an NTFS file system by storing varbinary(max) binary large object (BLOB) data as files on the file system". You can read more about FileStream here .

Answer 2

You seem to be doing a lot of extra, unnecessary work - somewhere along the lines, something goes wrong.

I tried your scenario with this code here and it works without a hitch - no problems at all, even for text files over 500 KB in size:

// read the whole text of the file in a single operation
// no filestream, memorystream and other messy stuff needed!
string file_content = File.ReadAllText(sourceFile);

Int32 file_length = content.Length;

// best practice: always put SqlConnection and SqlCommand into using() {..} blocks!
using (SqlConnection _con = new SqlConnection(CS))
{
   string _query =
      "INSERT INTO Page_File_Store (file_length, file_path, file_string) " + 
      "VALUES (@file_length, @file_path, @file_string)";

   using(SqlCommand _cmd = new SqlCommand(_query, _con))
   {
       // just add the three parameters with their values 
       // ADO.NET figures out the rest (datatypes etc.) by itself!
       _cmd.Parameters.AddWithValue("@file_length", file_length);
       _cmd.Parameters.AddWithValue("@file_path", fileName);
       _cmd.Parameters.AddWithValue("@file_string", content);

       _con.Open();
       _cmd.ExecuteNonQuery();
       _con.Close();
   }
}

That should definitely work - try it!

Marc

Answer 3

Does it solve your problem if you replace this:

SQL_File_Load.Parameters.Add(new SqlParameter("@file_string", content));

With this:

SqlParameter contentParameter = new SqlParameter("@file_string", SqlDbType.NVarChar, -1);
contentParameter.Value = content;
SQL_File_Load.Parameters.Add(contentParameter);

The -1 is interpretted as (MAX)