在Hadoop DistributedCache上存储TreeSet

Question

I am trying to store a TreeSet on a DistributedCache for use by a Hadoop map-reduce job. So far I have the following for adding a file from HDFS to a DistributedCache :

Configuration conf = new Configuration();
DistributedCache.addCacheFile(new URI("/my/cache/path"), conf);
Job job = new Job(conf, "my job");
// Proceed with remainder of Hadoop map-reduce job set-up and running

How do I efficiently add a TreeSet (that I already have built in this class) to this file that I am adding to the DistributedCache? Should I be using Java's native serialization to somehow serialize this onto the file?

Note that the TreeSet is built once in the main class that starts the map-reduce jobs. The TreeSet will never be modified and I simply want every mapper to have read-only access to this TreeSet without having to rebuild it over and over.

Answer 1

Serializing the TreeSet seems to be the approach. You do not need to create a HashMap in this case. Just deserialize the TreeSet from the file and use the methods to search based on the key. I like this approach.