Need help choosing database server

Question

Good day everyone.

Recently I was given a task to develop an application to automate some aspects of stocks trading. While working on initial architecture, the database dilemma emerged. What I need is a fast database engine which can process huge amounts of data coming in very fast. I'm fairly experienced in general programming, but I never faced a task of developing a high-load database architecture. I developed a simple MSSQL database schema with several many-to-many relationships during one of my projects, but that's it.

What I'm looking for is some advice on choosing the most suitable database engine and some pointers to various manuals or books which describe high-load database development.

Specifics of the project are as follows:

• OS: Windows NT family (Server 2008 / 7)
• Primary platform: .NET with C#
• Database structure: one table to hold primary items and two or three tables with foreign keys to the first table to hold additional information.
• Database SELECT requirements: Need super-fast selection by foreign keys and by combination of foreign key and one of the columns (presumably DATETIME)
• Database INSERT requirements: The faster the better :)

If there'll be significant performance gain, some parts can be written in C++ with managed interfaces to the rest of the system.

So once again: given all that stuff I just typed, please give me some advice on what the best database for my project is. Links or references to some manuals and books on the subject are also greatly appreciated.

EDIT: I'll need to insert 3-5 rows in 2 tables approximately once in 30-50 milliseconds and I'll need to do SELECTs with 0-2 WHERE clauses with similar rate.

Answer 1

The technology is irrelevant. This is about hardware.

Need super-fast selection by foreign keys and by combination of foreign key and one of the columns (presumably DATETIME)

So you need something like a supermicro 72 disc computer system filled with SSD's and 256gb memory as cache. Simple as that. Addd processors as needed.

Problem is that it is EXPENSIVE, but at the end databases are about IO. Point.

Alternatively go with a click stream database (KD+) and intead of paying 30.000 USD for hardware they ask for 20.000 USD For the smallest software license to start.

I know this sounds hard, but adatabase speed - unless you do stupid things that kill performance - IS about hardware. PRettym uch the IOPS budget (independent IOPS per second). And not about software.

Answer 2

That list of yours are just basic requirements. You can use almost any database for that, although I wouldn't recommend MySQL. It can slow down a little when you do many inserts, because this clears you query cache. Other DBMSs handle this better.

But then again. Three tables... you could almost do it in Access.

What you didn't specify is the amount of reads and inserts you expect. Our poor little MySQL database runs millions of queries a day for our website. Only a small part of this are inserts and updates (DML statements), so MySQL handles this very well, but the amount of queries, their complexity and the amount of data involved are more important data than the number of tables.

Answer 3

Where you already have experience with Sql Server, I would suggest sticking with it in some form. Also, Sql Server and .NET have a nice synergy going (obviously) because they are both Microsoft products.

Answer 4

If it doesn't have to be SQL, maybe a system like Cassandra is a good option. These kind of databases are used by high frequency websites like social media. Don't know how it installs under Windows server, though.