hadoop会比MySQL快吗

Question

I am facing a big data problem. I have a large MySQL (Percona) table which joins on itself once a day and produces about 25 billion rows. I am trying to to group together and aggregate all the rows to produce a result. The query is a simple join:

--This query produces about 25 billion rows
SELECT t1.colA as 'varchar(45)_1', t2.colB as 'varchar(45)_2', count(*)
FROM table t1
JOIN
table t2
on t1.date = t2.date
GROUP BY t1.colA, t2.colB

The problem is this process takes more than a week to complete. I have started reading about hadoop and wondering if the map reduce feature can improve the amount of time to process the data. I noticed HIVE is a nice little add-on to allow SQL like queries for hadoop. This all looks very promising, but I am facing an issue where I will only be running on a single machine:

6-core i7-4930K
16GB RAM
128 SSD
2TB HDD 

When I run the query with MySQL, my resources are barley being used, only about 4Gb of ram and one core is only working 100% while the other are working close to 0%. I checked into this and found MySQL is single threaded. This is also why Hadoop seems to be promising as I noticed it can run multiple mapper functions to better utilize my resources. My question remains is hadoop able to replace MySQL in my situation in which it can produce results within a few hours opposed to over a week even though hadoop will only be running on a single node (although I know it is meant for distributed computing)?

Answer 1

Some very large hurdles for you are going to be that hadoop is really meant to run on a cluster and not a single server. It can make use of multiple cores but the amount of resources that it will consume will be very significant. I have a single system that I use for testing that has hadoop and hbase. It has namenode, secondary name node, data node, nodemanager, resourcemanager, zookeeper etc running. This is a very heavy load for a single system. Plus HIVE is not a true SQL compliant replacement for a RDBMS so it has to emulate some of the work by creating map/reduce jobs. These jobs are considerably more disk intensive and use the hdfs file system for mapping the data into virtual tables (verbage may vary). HDFS also has a fairly significant overhead due to the fact that the filesystem is meant to be spread over many systems.

With that said I would not recommend solving your problem with Hadoop. I would recommend checking out what it has to offer though in the future.

Have you looked into sharding the data which can take advantage of multiple processors. IMHO this would be a much cleaner solution.

http://www.percona.com/blog/2014/05/01/parallel-query-mysql-shard-query/

You might also look into testing postgres. It has very good parallel query support built in.

Another idea is you may look into trying an olap cube to do the calculations and it can rebuild the indexes on the fly so that only changes will be taken into affect. Due to the fact that you are really dealing with data analytics this may be an ideal solution.

Answer 2

Hadoop is not a magic bullet.

Whether anything is faster in Hadoop than in MySQL is mostly a question of how well your abilities to write Java code (for mappers and reducers in Hadoop) or SQL are...

Usually, Hadoop shines when you have a problem running well on a single host, and need to scale it up to 100 hosts at the same time. It is not the best choice if you have a single computer only; because it essentially communicates via disk . Writing to disk is not the best way to do communication. The reason why it is popular in distributed systems is crash recovery. But you cannot benefit from this: if you lose your single machine, you lost everything, even with Hadoop.

Instead:

1. figure out if you are doing the right thing . There is nothing worse than spending time to optimize a computation that you do not need. Consider working on a subset, to first figure out whether you are doing the right thing at all... (chances are, there is something fundamentally broken with your query in the first place!)

2. optimize your SQL. Use multiple queries to split the workload. Reuse earlier results, instead of computing them again.

3. reduce your data. A query that is expected to return 25 billion must be expected to be slow! It's just really inefficient to produce results this size. Choose a different analysis, and double-check that you are doing the right computation; because most likely you aren't; but you are doing much to much work.

4. build optimal partitions. Partition you data by some key, and put each date into a separate table, database, file, whatever, ... then process the joins one such partition at a time (or if you have good indexes on your database, just query one key at a time)!

Answer 3

Yes you are right MySQL is single threaded ie 1 thread per query.
Having 1 machine only I don't think it will help you much because you may utilize the cores but you will have contention over I/O since all threads will try to access the disk.
The number of rows you mentioned are a lot but you have not mentioned the actual size of your table on disk.
How big is your table actually? (In bytes on HD I mean)
Also you have not mentioned if the date column is indexed. It could help you if you removed the t2.colB or removed the GROUP BY all together.
GROUP BY does sorting and in your case it isn't good. You could try to do the group by in your application.
Perhaps you should tell us what exactly are you trying to achieve with your query. May be there is a better way to do it.

Answer 4

I had a similarly large query and was able to take advantage of all cores by breaking up my query into multiple smaller ones and running them concurrently. Perhaps you could do the same. Instead of one large query that processes all dates, you could run two (or N) queries that process a subset of dates and write the results into another table.

ie if your data spanned from 2012 to 2013

SELECT INTO myResults (colA,colB,colC)
SELECT t1.colA as 'varchar(45)_1', t2.colB as 'varchar(45)_2', count(*)
FROM table t1
JOIN table t2 on t1.date = t2.date
WHERE t1.date BETWEEN '2012-01-01' AND '2012-12-31'
GROUP BY t1.colA, t2.colB

SELECT INTO myResults (colA,colB,colC)
SELECT t1.colA as 'varchar(45)_1', t2.colB as 'varchar(45)_2', count(*)
FROM table t1
JOIN table t2 on t1.date = t2.date
WHERE t1.date BETWEEN '2013-01-01' AND '2013-12-31'
GROUP BY t1.colA, t2.colB