从Sql迁移到NoSql数据库的限制

Question

We are facing performance related issues in our current MySQL DB. Our application is pretty heavy on a few tables ~20. We run lot of aggregation queries on this table as well as writes. Most of our teams are developers and we don't have access to a dba which might help in retuning our current db and make things work faster.

Moving to NoSql is an option. But seriously thinking what are the higher limits in terms of

• Volumes (Current volumes per day ~50GB)
• Structured or Raw Data? (Structured Data)
• IO stats on DB - ( Current rate is 60 KB/Sec)
• Record writes - (now 3000 rows/sec)

Question arise

• Is 50GB is high enough to consider NoSql? Some documentation recommends more than a TB
• The data should be raw data, which can be further processed to get structured and use in application
• MySql scales out at 3000 rows/secs, not sure MySql can be further tuned

HBase seems to be promising for Analytic application.

Would like to get some guidelines on limits of RDBMS one can think of moving to NoSQL

Answer 1

This is such a broad topic so don't believe there are any "right" answers but maybe a few general recommendations would help:

I think you should think of this challenge in terms of picking the right tool for the problem. All databases have their pros and cons and in some challenges the best approach is to use an entire toolbox to get the job done.

Note that moving your data, or even just parts of it, to different datastores is rarely a non-trivial effort. Use this chance to rethink about your data model before implementing it.

Getting this job done should also take into account more requirements, such your growth plans for example. It looks you're at this crossroads because your original assumptions->choices are no longer en par with reality. If you want to delay the next time you're at the same place, you should use this opportunity to do so.

Lastly keep in mind that the job really done only after you do something with all that captured data - or else I'd recommend you use the infinitely-scalable write-to-/dev/null design pattern ;) Put differently, unless your data is write-only, you'd want to make sure that whatever SQL/NoSQL/NewSQL/other datastore that you choose can also get you the data/information/knowledge inside your use case's acceptable time frames.

Answer 2

It will probably worth it given your current infrastructure, but keep in mind that it's going to be a huge task , since you're going to need to redesign the whole process . HBase can help you, as it has some neat features, like realtime counters (which in some cases eliminates the needing of periodic rollups), or per-client buffering (which can allow you to scale to the >100k writes per second), but, be warned it cannot be queried in the same way you query a relational database, so, you're going to need to carefully plan it to make it work for you.

It seems that your main issue is with the raw data writes, sure, you can definitely rely on HBase for that, and then do the rollups every X min to store the data in your RDBMS so it can be queried as usual. But given you're doing them every minute, which is a very short gap, why don't you keep the data in memory and flush it the rolled up tables every minute?. Sure, you could loss data, but I don't know how critic is for you loosing one minute of data, and that alone could help you a lot.

Anyway, the best advice I can think of: read a book, understand how HBase works first, dig into the pros & cons, and think about how it can suit your specific needings. This is crucial because a good implementation is what is going to determine if it's a success or a total failure.

Some resources:

HBase: The Definitive Guide

HBase Administration Cookbook

HBase Reference guide (free)