在数据库中存储大量数据

Question

I have question regarding the storage of large amount of data. The situation is the following:

1. I want to store

   • GPS Coordinates(latitude and longitude) (every minute or even less interval, but I'm considering every minute)
   • Event, which can be repeated for several coordinates
   • Datetime or timestamp of entry(dunno which is better to use in my case)
   • (user id)

2. I want to be able to query:

   • Event by zone(defining the range of latitude and longitude, for example from (1,1) to (2,2))
   • User tracking from date X to date Y (one or more users)

So far I was thinking on the solution:

Solution 1

id_user (int)
id_experince (int)
id_event (int)
dt (datetime)
latitude (decimal)
longitude (decimal)

I started to do some calculations and that would be something like: - around 500 entries per day/user - since I'm preparing application for some load, there can be around 100-150 users, which will be 75000 entries/day - after one month there will be millions of entries

Probably, Solution 1 is not good solution, since the size of database with grow very fast.

Solution 2

Have 2 tables, one of which will be aggregate coordinates according to event, for example I have event "dinner" and it takes 30 minutes, so 30 entries will be grouped in one field with BLOB type. This table will look like:

id_user (int)
id_experience (int)
id_event (int)
dt (datetime)
coordinates(blob)

And another table, which have have calculated locations with some "width" and "length", having pointer to the first table

latitude (decimal)
longitude (decimal)
id_entry_in_first_table (int)

This solution only partially solves my problem, imagine, that some events will not be more several minutes and there is a need for the second database..

Solution 3

This is probably not very correct solution, but it seems making some sense. I have user associated with some kind of experience, which has start date and end date. When experience adds, I will create dump of data for that experience and save to the file, deleting the entries related to the experience. When the user will want to consult "archived" experience, I will load data into some temporary table and drop it within one day(for example), in this case I will save the data according to the solution 1.

The main question is: are any of the presented solutions acceptable in terms of performance of the database? Is there any better solution for my problem?

Answer 1

"Millions of entries" sounds like a lot, but this is what databases are designed to handle. However you design it, if you optimise it according to how you want to extract results from it later (as thats what will take the time as opposed to the inserts) then you're good to go.

Saying that of course... if you have lots of users doing lots of things at the same time to your database then I think your server/bandwidth with go before your database does!

Answer 2

I would choose a master detail approach.

Two advantages:

1. Yo don't have redundant entries (1 master row and x child rows with coordinates)

2. It is still easy to query (in contrast to the blob approach).

    SELECT m.id_user, m.id_experince, m.id_event, c.latitude, c.longitude FROM master_table m LEFT JOIN child_table c ON m.id = c.master_table_id 

And this should be pretty fast even with many millions of records in the master table, if you setup a foreign key or index on master_table_id

Answer 3

You probably want to read this: http://dev.mysql.com/doc/refman/5.0/en/spatial-extensions.html .

Broadly speaking, as long as you can use indexes in your queries, huge tables aren't an issue - billions of records can be queried on consumer grade laptops. You should have an archiving strategy if you intend to scale to huge numbers of historical records, but it's not a huge priority.

Far more tricky is to support your desire to find events within a certain geographic boundary; it's easy for this to break your indexing strategy in all sorts of nasty ways. If you have to query based on mathematical operations, you may not be able to use an index - so finding users within a radius of a 1 mile circle might have to evaluate the circle formula for every record in your database table.

The spatial extensions offer a solution for this - but they're not "free", you have to optimize your design specifically for this.