Storing great amount of data in ASP.NET

Question

I've built a website in ASP.NET (Umbraco CMS) which has a contest module. The company launches about two contests every month and each contest has an average of ~15.000 contestants. These contestants are stored in a database tables which is designed like this:

Contestant

Id
Name
ZipCity
Address
Country
Email
DateAttended

ZipCity

Id
Zipcode
City

ContestAnswer

Id
AnswerNodeId (Umbraco node id)
ContestNodeId (Umbraco node id)
ContestantId
Date

Each contest can have one question and N number of answer options.

The way I have created it to save the number of records in the database is that for every single answer, I check to see if there's already a contestant with the given e-mail in the Contestant table. If there is, I use that one and if not, I create a new contestant. Also, for every contestant I check if the provided zipcode is already in the ZipCity table and create a reference to that.

Even though I have done it this way instead of blindly creating a new Contestant for every contest, the database keeps getting filled up :-( I've increased the capacity twice now and still getting the same exception after some time:

Could not allocate space for object ... because the filegroup is full.

The company wants to keep the contestants for each contest they launch, so deleting records after a contest has ended is not an option it seems :-/

So, now I've been wondering if there is a smarter way of storing these large amount of data? Thoughts on this?

Any inputs are greatly appreciated!

Answer 1

We don't know much about your hard drive setup.

http://blog.sqlauthority.com/2009/05/31/sql-server-create-multiple-filegroup-for-single-database/

http://msdn.microsoft.com/en-us/library/bb522469.aspx

The first link will explain the concept.

The second link will show the syntax.

The 3 items you want to look at are:

[ , SIZE = size [ KB | MB | GB | TB ] ] 
[ , MAXSIZE = { max_size [ KB | MB | GB | TB ] | UNLIMITED } ] 
[ , FILEGROWTH = growth_increment [ KB | MB | GB | TB| % ] ] 

Why?

The first one is the initial size. The second one is the maximum size. And IF THIS IS SET, even though your harddrive may have space on it, the database will cease to grow. The third one is "how big will I expand, if I have not violated the maxsize rule".

Now. You can set the MAXSIZE to UNLIMITED.

HOWEVER. You'll start to get errors when your hard drive fills up.

So you gotta play dba, and pick which option is best.
Personally, I would set the MAXSIZE to something less than the free space on the drive, so you don't hit the edge point.

ALSO: You can create different file groups, and put different different tables on different file groups. This is handy if you know one table will be HUGE, but the other will be small.

So...check to see if you have a MAXSIZE setting set already.

Then you gotta play roulette with your tables and harddrives and filegroups and tables.

You can also put your log files on a different hard drive as another option for space issues.

Here is a good little sample of creating file groups, and then moving tables over to that new file group.

http://www.daveturpin.com/2010/09/move-tables-to-new-filegroup/

Below is a script that I have for creating a database using SQLCMD Mode. Note the directories must "pre exist" (on the server, not your local machine if you are aren't actually sitting on the sql server).

:Setvar DBName MyFirstDatabase01
:Setvar DataFilesBaseDirectory "C:\WUTemp\Some Folder\"
:Setvar LogFilesBaseDirectory "C:\WUTemp\Some Folder\"
:Setvar DatabasePrimaryDataFileStartSizeMB 9
:Setvar DatabasePrimaryDataFileGrowthMB 8
:Setvar DatabasePrimaryLogFileStartSizeMB 7
:Setvar DatabasePrimaryLogFileGrowthMB 6




Use [master];
GO


if exists (select * from sysdatabases where name='$(DBName)')
BEGIN
        DROP DATABASE [$(DBName)];
END

GO



--Create Database $(DBName)
--G--O


DECLARE @device_directory_data NVARCHAR(520)
DECLARE @device_directory_log NVARCHAR(520)
--SELECT @device_directory_data = SUBSTRING(filename, 1, CHARINDEX(N'master.mdf', LOWER(filename)) - 1) FROM master.dbo.sysaltfiles WHERE dbid = 1 AND fileid = 1
--SELECT @device_directory_log = SUBSTRING(filename, 1, CHARINDEX(N'master.mdf', LOWER(filename)) - 1) FROM master.dbo.sysaltfiles WHERE dbid = 1 AND fileid = 1



select @device_directory_data = '$(DataFilesBaseDirectory)'
select @device_directory_log = '$(LogFilesBaseDirectory)'

print  @device_directory_data
print @device_directory_log


EXECUTE (N'CREATE DATABASE $(DBName)
  ON PRIMARY (NAME = N''$(DBName)'', FILENAME = N''' + @device_directory_data + N'$(DBName).mdf'', SIZE = $(DatabasePrimaryDataFileStartSizeMB) MB, FILEGROWTH = $(DatabasePrimaryDataFileGrowthMB)MB)
  LOG ON (NAME = N''$(DBName)_log'',  FILENAME = N''' + @device_directory_log + N'$(DBName).ldf'', SIZE = $(DatabasePrimaryLogFileStartSizeMB) MB, FILEGROWTH = $(DatabasePrimaryLogFileGrowthMB)MB)')

DECLARE @logsize char(1)
SELECT @logsize = CASE WHEN convert(varchar, Serverproperty('ProductVersion')) LIKE '9%'
                       --THEN '10'
                       --ELSE '5'
                       THEN '13'
                       ELSE '14'

                  END


EXECUTE(N'ALTER DATABASE $(DBName)
    ADD LOG FILE (NAME = N''$(DBName)_log2'',
                  FILENAME = N''' + @device_directory_log + N'$(DBName)log2.ldf'', SIZE = ' + @logsize + ' MB)')


-- Make the database case sensitive to clean up the development effort
--  EXECUTE(N'ALTER DATABASE $(DBName) COLLATE SQL_Latin1_General_CP1_CS_AS')  




exec sp_dboption '$(DBName)','trunc. log on chkpt.','true'
exec sp_dboption '$(DBName)','select into/bulkcopy','true'
GO




GO

Answer 2

There is not much you can do in terms of reducing space that will cause the data requirements to go down. Also, 15k per month is not much really unless you are on some shared sql hosting that has severe limitations. Anyway, not much you can do here unfortunately.

Perhaps you can try with enabling sql server compression. Check these articles for more details

http://www.bradmcgehee.com/2010/03/an-introduction-to-data-compression-in-sql-server-2008/

http://msdn.microsoft.com/en-us/library/cc280449.aspx