如何存储实时数据

Question

I am looking for a way (database ? data file ?) to store a large amount of real time data. There can be 20,000 points each seconds (time + 32bits data). To avoid 20K samples per seconds, I can buffer it (because of 32bits, it would not be too much memory consumming to buffer).

I then must be able to read it while still recording data. I saw that kdb+ would be a good fit, but the solution must be open source and work with C# and run on the Windows OS.

Thanks.

Answer 1

Look at CEP (complex event processing) systems like Microsoft StreamInsight . Here is developer guide.

Answer 2

You are dead. Point. There IS NO open source solution that does what you want. The requirements of open source + c# basically kill all available solutions.

MySQL has SOME databases that are good for time series, sort of (storage systems). You could try thsoe.

That said, tuning SQL Server to accept 200.000 inserts is not that hard. I do 80.000 now and I know I have issues on the cpu timing side (virtual machine) that I will soon resolve, my discs are not busy, nor is the CPU.

Answer 3

Any time you hit the disk, you'll have a hard time keeping up with that amount of events. Like one of the previous posters said, you should look at a CEP system such as StreamInsight. If you are a C# developer, SI will be a good fit.