使用Java进行云中的重复数据删除

Question

I am trying to implement a data deduplication program in the cloud using Java.

I'm not sure how to proceed with the implementation.

First, I wanted to do a simple file compare of the file size, date and name of the file. However, this is ineffective since the file might have same content but a different name.

I have decided on a simple algorithm which is file upload -> file chunking -> Rabin-karp hashing -> determine to see whether can upload file.

Will this be fine or are there any improvements?

Where would I be able to find out more information on this? I have tried looking around the Internet but I can't find anything. Most of it is just broken down into certain implementations but without explanation or details on file chunking or Rabin-karp hashing.

I would want to know about which Java libraries I should look into regarding this program.

Answer 1

It would be easier if you state your problem constraints. Assuming the following:

• The smallest indivisible unit of data is a file
• Files are reasonably small to fit in memory for computing hashes
• Your files are in some cloud bucket or something where you can list them all. Also that eliminates identical filenames.

You can probably narrow down your problem.

1. Iterate through all the files in all the files using some fast hashing algorithm like a basic CRC checksum and build a map. (Can be easily parallelized).
2. Filter out all the files which have a collision. You can easily leave out the rest of the files which for all practical purposes should be a pretty reasonable chunk of the data.
3. Run through this remaining subset of files with a cryptographic hash (or worst case, match the entire files) and identify matches.

This can be refined depending on the underlying data.

However, this is how I would approach the problem and given the structure of it; this problem can be easily partitioned and solved in a parallel manner. Feel free to elaborate more so that we can reach a good solution.