CouchDB Compression

Question

I have a MySQL database that I wanted to store on an external 4TB drive, but when I copied the database over to it, the server failed to restart. I found out it had something to do with a sector size issue.

This gave me reason to make the jump to NoSQL. I like CouchDB for its ease of use and HTTP API, but the database is simply not going to work for me without compression. I have a 40GB MySQL database, and the data migration isn't even a tenth of the way complete and it's already over 100GB.

Is there something I'm missing? Do/Can I enable compression?

Thanks!

Answer 1

CouchDB trades disk-space for read/write speed. It's very likely a comparable CouchDB database will take up more disk-space than MySQL.

That being said, there are a number of things you can do to conserve disk-space:

• Database file compression is available, (as you pointed out) but you should probably experiment with the various algorithms to find out what works best in practice.
• Database compaction periodically throughout your import process, and as part of routine maintenance.
• Lastly, when writing your views, do not emit the entire document as part of the index. Instead, use the include_docs=true query-param. (see docs for other params)

In other words, avoid this:

function (doc) {
  emit(doc.key, doc);
}

For each view you write like this, each emit means that the document is being duplicated in your database. Thus, you only need to do this: (most of the time, you don't need that 2nd argument)

function (doc) {
  emit(doc.key);
}

I'm sure there are other things you can do, if I think of more I'll amend this answer. (please comment if you know of anything I missed)