高性能大文本存储解决方案

Question

Here is the scenario:

1) All the data currently is stored in SQL Server

2) There is one table X that stores the most important and highly accessed data

3) Each record in the table X has one column Y that stores up to 2GB of text

4) Number of records is growing (> 10 million and growing)

5) The column Y with max of 2GB of text per row takes about 90% of space of table X

Problem: database becomes unmanageable and we need to store data from Column Y somewhere else (outside of SQL Server)

Question:

What is the best solutions for storing large text?

I am looking for a solutions with least amount of overhead :

• fast reads
• fast writes
• scalable
• can handle concurrency well

I don't care about full text index or any extra overhead. Just plain storage and access.

PS

Some ideas that I have so far:

• very simple NOSQL database
• flat file - based database
• custom solution that I can write from scratch using some library

Answer 1

From what I read about NoSQL technologies, Riak may be a great fit, however, there's certainly overhead with this. Flat files sound promising, but highly OS and patch dependent - you don't want to be stuck explaining to a customer that Windows Updates caused performance problems.