使用 hadoop + mapreduce 在 couchdb 中处理数据

Question

I have a very very large quantity of data in CouchDB, but I have very recently found out how crippled the mapreduce functions in couch are (no chaining).

So I had this idea of running map reduce queries from the CouchDB database using Hadoop, and hopefully storing the final result in another CouchDB database?

Is this too crazy? I know I can set up Hbase to do this, but I do not want to migrate my data from CouchDB to Hbase. And I love couch as a data store.

Answer 1

Apparently CouchDB is supposed to be able to stream data to Hadoop via Sqoop , but I didn't see any other information than that link. Worst case, you can write your own input reader to read from CouchDB, or export your data regularly and throw it onto HDFS and run it from there.

Answer 2

The MapReduce functions in CouchDB are constrained to simplify caching of the results. Rather than having to search for views that are impacted by a change, views were designed to be self-contained.

This means that if you have complex MapReduce code, you can use a tool like CouchApp to embed functions within a MapReduce function. I'm having trouble finding the reference for this, but you the macro !code to embed JavaScript functions in views. Using require() or // !json, !code in CouchDB?

This could help to get some of the productivity benefit of chaining without chaining, by putting most of the code in shared functions, and merely calling the function in the different views. For the performance benefit of chaining, if that's what you're after, you may be better off just moving to HBase.