Has anyone used HBase, Tornado, Cassandra or HipHop with Drupal?

Question

I have a Drupal application that has a very large recordset, almost half a million nodes (475,181). Even simple joins are becoming too time consuming (3-10s), and we are becoming increasingly dependent on Memcached. I wonder if anyone has used the same technologies that Facebook is using with Drupal.

Answer 1

To answer your question: Not yet, but there is significant work being done on HipHop and a nosql database named MongoDB. More on those below.

As others have mentioned I would first make absolutely sure that your database tables are optimized properly, indexed and that your database has sufficient resources. If your database is choking on under 500k rows it's likely that there is something wrong. (We have over 150k node, about 600k rows in our node_revisions table and because the nodes have multiple images over a million rows in some of our cck tables.) I'm no MySQL expert and was able to get the database query times to under a hundred milleseconsds for most of our queries.) Here are the steps I would take before looking into Switching to a different db engine. (from easiest to hardest)

1. look for places that you need indexes. The most likely places are tables that have nid, vid or where there are a lot of queries that use a field value in the where, and any of the columns in your joins. This is where I got 80% of the improvement. Adding too many indexes can slow down inserts and deletes, but they are very simple to add, and remove.
2. Consider switching to INNODB it's almost always faster than the standard drupal 6 myisam. This is a super easy change.
3. Confirm that your my.cnf is set up properly. If you haven't changed it, it's probably wrong. The settings that ship with most distributions of MySQL are very, very conservative, and probably wrong for your environment.
4. Add RAM, if you have switched to INNODB adding RAM can make a huge difference in your performance, even without MySQL loves RAM.
5. Switch to pressflow if you can. It fixes a number of things that slow down regular drupal.
6. If you are still having problems look to moving away from CCK and created your own content types.
7. Review and optemize your views queries. Views isn't terribly efficient and you can sometimes make very big headway by modifying the queries.

Here is a great article on how much can be achieved with drupal and 1 single server.

If none of that works here's a few links regarding some of the newer haute technogogies that people are playing with. The most promising is MongoDB, and if you have the development resources to use it on a project I envy you. (It's still a little new new and unpolished for the small shop we are, but I can't wait to sink my teeth into it.)

drupal and hiphop

drupal and Mongodb - there is also a great talk in the sessions from drupalcon SF (last year)

Answer 2

You do realize HipHop and Tornado aren't database-software, right?

However, the db is almost like the core of a CMS, so when you have to change it to another system (well, actually even more, SQL to NoSQL) you should consider building a new / using another CMS rather than editing the Drupal-core.

Answer 3

我没有关于该主题的经验，但是https://drupal.stackexchange.com/questions/183/real-world-experience-in-scaling-and-tuning-performance确实看起来充满了有用的信息。