如何在hadoop mapreduce中将文件分区为较小尺寸以执行KNN

Question

In KNN like algorithm we need to load model Data into cache for predicting the records.

Here is the example for KNN.

So if the model will be a large file say1 or 2 GB we will be able to load them into Distributed cache. Example: Inorder to predict 1 otcome, we need to find the distnce between that single record with all the records in model result and find the min distance. So we need to get the model result in our hands. And if it is large file it cannot be loaded into Distributed cache for finding distance.

The one way is to split/partition the model Result into some files and perform the distance calculation for all records in that file and then find the min ditance and max occurance of classlabel and predict the outcome.

How can we parttion the file and perform the operation on these partition ?

ie  1 record <Distance> file1,file2,....filen
     2nd record <Distance> file1,file2,...filen

This is what came to my thought.

Is there any further way.

Any pointers would help me.

Answer 1

I think the way you partitionin the data mainly depends on your data itself.

Being that you have a model with a bunch of rows, and that you want to find the k closes ones to the data on your input, the trivial solution is to compare them one by one. This can be slow because of going through 1-2GB of data millions of times (I assume you have large numbers of records that you want to classify, otherwise you don't need hadoop).

That is why you need to prune your model efficiently (your partitioning) so that you can compare only those rows that are most likely to be the closest. This is a hard problem and requires knowledge of the data you operate on.

Additional tricks that you can use to fish out performance are:

• Pre-sorting the input data so that the input items that will be compared from the same partition come together. Again depends on the data you operate on.
• Use random access indexed files (like Hadoop's Map files) to find the data faster and cache it.

In the end it may actually be easier for your model to be stored in lucene index, so you can achieve effects of partitioning by looking up the index. Pre-sorting the data is still helpful there.