Will reducer out of java heap space

Question

I am implementing a program using Hadoop. My question is how to deal with java out of space problem, I added some property configuration into xml file, but it didn't work. Increasing number of reducers doesn't work for me either. Because in my program every reducer needs large sparse whole matrix, and I am not allowed to change this logic. Yet every reducer will receive an entry with column id as key, and column vector as value.Is there any way I can get out of this dilemma?

PS:I was first thinking that reducer will take column one by one, and that won't cause out of memory issue. But it seems not this way. Actually, I am trying to implement an algorithm described in a paper. The algorithm for reducer is listed below. Line 11 is the column I mentioned, and each reducer has every column of the matrix.

Answer 1

The reason why you use something like hadoop is because you cant fit the entire data set into memory. Either you don't change the logic and try to find a computer that's big enough or you parallel-ize the algorithm and exploit hadoop.