Apache Giraph / Hadoop：通过自定义ArrayWritable进行迭代

Question

I thought this would be simple to implement, but it's starting to be a pain.

I've got a ArrayWritable subclass like so:

public class VertexDistanceArrayWritable extends ArrayWritable {
    public VertexDistanceArrayWritable() {
        super(VertexDistanceWritable.class);
    }
    public VertexDistanceArrayWritable(VertexDistanceWritable[] v) {
        super(VertexDistanceWritable.class, v);
    }
}

And a Writable subclass like so:

public class VertexDistanceWritable implements Writable {

    //Implements write, readFields, and some custom functions that aren't used yet

}

In my Giraph compute function, Messages are VertexDistanceArrayWritable's. I want to iterate through every VertexDistanceWritable every message (VertexDistanceArrayWritable). Here is my compute function:

@Override
public void compute(Vertex<Text, MapWritable, FloatWritable> vertex,
    Iterable<VertexDistanceArrayWritable> messages) throws IOException {

    for(VertexDistanceArrayWritable message : messages) {
        for(VertexDistanceWritable distEntry : message) {
            //Do stuff with distEntry
        }
    }

    //do other stuff

    vertex.voteToHalt();
}

When I compile the code, I get this error:

for-each not applicable to expression type
    for(VertexDistanceWritable distEntry : message) {

  required: array or java.lang.Iterable
  found:    VertexDistanceArrayWritable

So now I have a problem. I want to iterate over the arrayWritable sub-class.

I've tried the following:

• Change that line to for(VertexDistanceWritable distEntry : message.toArray()) which tells me that for-each not applicaable to type Object (required: array or java.lang.Iterable, found: Object).

• Change that line to for(VertexDistanceWritable distEntry : message.get() ) , which gives me error: incompatible types -- required: VertexDistanceWritable, found: Writable . This is the strangest problem -- VertexDistanceWritable extends Writable, shouldn't this work fine?

• Writing my own custom "get_foo()" function for VertexDistanceWritable, which returns values as a VertexDistanceWritable[]. Of course, values is private, and has no getter function according to the documentation other than get() which I'm already having problems with

I just want a way to iterate over my VertexDistanceArrayWritable class. Is this even possible in Hadoop? It has to be, right? I should be able to iterate over a bunch of elements I made in an array, no? It seems like pretty darn basic stuff.

Answer 1

After about 30 minutes of experimenting and googling, I found out a clue here . Sort of cheesy, but it seems to compile correctly. Basically just use a Writable then cast it to my custom writable.

for(VertexDistanceArrayWritable message : messages) {
    for(Writable distWritable : message.get()) {
        vertexDistanceWritable distEntry = (VertexDistanceWritable) distWritable;
        //do other stuff
    }
}

I haven't yet confirmed if it works correctly, I will update and confirm my answer when I can make sure it works.

edit: it works. Might require a copy constructor since I had one for VertexDistanceWritable, but never checked that out.