Mysql: Millions of Records in table with 10 million updates in a day

Question

We Have a case where we have to record millions of status updates in a day on a table having records up to the tune of 146000000. I am not sure if MySQL is up to it. Here is the complete scenario:

1. On day one Table will have 1 million records which will eventually grow to up to 150 million in 2 years.
2. At a given point in time at max 1.4 million records will be live ie even after we have 150 million records.
3. Records should not grow beyond 150 millions as we look to archive data older than 2 years.
4. The live 1.4 million records will get updates for their statuses which we need to update in the same table. The updates will be up to the tune of 20 million a day on these 1.4 million records.
5. I would be OK to not have any foreign key constraints on the table if they come in way of so many updates in the table.

We are using MySQL 5.5.

My concern and question is - will MySQL be able to hold upto our requirements out of the box (I have a feeling that we may see deadlocks when updates would be happening at above mentioned pace)? And in case it can't, what should we do that we can build all that is said above?

Thanks in advance,

Answer 1

我建议您必须使用任何nosql数据库。像cassandra相比，与mysql /相比，性能会提高30％到40％

Answer 2

Is it possible to split the table into multiple ones? Eg one table for each hour/day/week/month

If yes you can create a separate table for the desired time inteval and store the updates separately.

Or you can try to separate the data into multiple tables by another sign eg by customer.

It's hard to advice without more info.

Are you goint to gather some statisticas from the table(s)?

Answer 3

There are a lot of nosql databases available in the market for the same use case which you are having.

You can have all your problems resolved via that.

Answer 4

Oh my....

I still remember in 2002 when we replaced an Oracle database as the requirement was to insert/update 50M records a day and keep the records for at least 7 years with a need for simultaneous reporting/monitoring ;-) (one NMS application for mobile networks)

Before saying that go for NoSQL or NewSQL or what-so-ever, it would be helpful to understand the bigger picture. Do you f.ex. need real-time reporting? Replication? on-line backups and other features that are needed for mission critical applications?

How experienced your team is in DBMS technology? Have you workjed with anythin else but MySQL?

You mention dead-locks? Do you see them now? I would say, don't want to hurt anybody's feelings though, that the dead-locks are mostly due bad design! There are 100's of ways to avoid or swiftly resolve dead-locks starting from queuing, transaction encapsulation, transaction monitors, optimistic concurrency etc.

To me those numbers you provided aren't that high, but the performance can be ruined by design what ever DBMS you are using. Without understanding the full set of needs in ACID, SQL compatibility and other needs, I will NOT recommend jumping to XyzzySQL bandwagon WITHOUT fully understanding where the true requirements and related issues are.

So, in short: Your numbers aren't that worrying. I have seen bigger systems being implemented (using InnoDB = money). I have seen MySQL easily to digest with steady 24h load something like 15M inserts/updates mixed load a day with no hick-ups and tricks.

If your needs are relaxed, you could take the "black hole" or in-memory approach and write to non-permanent tables which would be replicated (over the time) to reliable storage within MySQL files. This is a good approach when it does not matter if you loose "few seconds or minutes" of inserts/updates.

The situation is very different if you have a steady laod with limited peaks or, as in many real-life applications, a huge peak load over certain hours of the day.

Cheers, //Jari