处理大量永久数据的最佳方法

Question

I'm developing a PC app in Visual Studio where I'm showing the status of hundreds of sensors that are connected via WiFi. The thing is that I need to hold on to the sensor data even after I close the app, so I'm considering some form of permanent storage. These are the options I've considered:

1) My Sensor object is relatively compact with only a few properties. I could serialize all the objects before closing the app and load them every time the app starts anew.

2) I could throw all the properties (which are mostly strings and doubles) into a simple text file and create a custom protocol for storage and retrieval.

3) I could integrate a database with my app. Someone told me this is the best way to go about it, but I'm a bit hesitant seeing as I'm not familiar with DBs.

Which method would yield the best results in terms of resource usage and speed? Or is there some other, better way to go about this?

Answer 1

First thing you need is to understand is your problem. For example, when the program is running do you need to have everything in memory at the same time or do you work with your sensors one at a time?

What is a "large amount of data"? For example, to me that will never be less than million (or billion in some cases).

Once you know that you shouldn't be scared of using something just because you are not familiar to it. Otherwise you are not looking for the best solution for your problem, you are just hacking around it in a way that you feel comfortable.

This being said, you have several ways of doing this. Like you said you can serialize data, using json to store and a few other alternatives but if we are talking about a "large amount of data that we want to persist" I would always call for the use of Databases (the name says a lot). If you don't need to have everything in memory at the same time then I believe that this is you best option.

I personally don't like them (again, personal choice) but one way of not learning SQL (a lot) while you still use your objects is to use an ORM like NHibernate (you will also need to learn how to use it so you don't get things a slower).

If you need to have everything loaded at the same time (most often that is not the case so be sure of this) you need to know what you want to keep and serialize it. If you want that data to be readable by another tool or organize in a given way consider a data format like XML or JSON.

Answer 2

Also, you can use mmap-file. File is permanent, and keep data between program run. So, you just keep your data structs in the mmap-ed area, and no more.

MSDN manual here: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366556%28v=vs.85%29.aspx

Answer 3

Since you need to load all the data once at the start of the program, the database case seems doubtful. The DB necessary when you need to load a bit of data many times.

So first two cases seem more preferred. I would advice to hide a specific solution behind an interface, then you'll can change it later.

Standard .NET serialization of sensors' array is more simple probably, and it will be easier to expand.