Backup Riak Bitcask backend

Question

I'm trying to understand how to properly backup Bitcask on a Riak node. I was under the impression that one of the strengths of Bitcask is the possibility to run filesystem backups while the node is running. Looking at the official documentation it says to always stop the node before doing backups, even when using the Bitcask backend.

I can see 3 ways of doing backups, ordered from easier to more complicated:

1. copy bitcask directory of a running node using tar, rsync or cp.

2. snapshot filesystem while node is running.

3. stop node, snapshot filesystem, start node.

So, would option 1 successfully backup bitcask, or will there be risk of data loss? if so, what about option 2?

Edit below. This is in response to Craig's answer:

Reading the Bitcask "whitepaper" it says there is compaction logic built in. However, I guess Riak might add its own merge logic on top of that.

http://basho.com/wp-content/uploads/2015/05/bitcask-intro.pdf

Also, the official documentation for Riak 1.4.12 only states that the node must be stopped when using the LevelDB backend, not Bitcask. Ofcourse, something might have changed in riak 2.0+.

http://docs.basho.com/riak/1.4.12/ops/running/backups/

And also, reading about the Bitcask backend in the latest official documentation gives me the impression that live backups are possible.

docs.basho.com/riak/kv/2.1.4/setup/planning/backend/bitcask/

If the node should be stopped due to merging and compaction. Is it perhaps safe to backup Bitcask on a running node outside of the merge window or with merge policy set to never?

I understand the documentation, but thought one of the neat things about Bitcask was live backups and it fitting nicely together with Riak's goal of "high availability". But it looks like that is not the case. At least not anymore.

Answer 1

The best advice for backing up Riak nodes is here:

http://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/backing-up/

While a normal Bitcask database (not associated with a Riak node) might be ok with online backups it is recommended that you stop a Riak node: "to prevent data loss as a result of the background merging and compaction processes of Riak's backends." In other words if you snapshot the node while it is running there is a good chance that some of the files will be corrupt on the snapshot.

There is work being done with the LevelDB backend that will allow running nodes to be backed up (and some organizations are already doing it with their own techniques). I don't know that the same functionality will be added for the Bitcask backend.