Java & Pig - Is it possible to get the pig script's output into a Java variable?

Question

I wrote a script in Pig Latin and I executed it through Java. Here is my code:

public static void main(String args[]){

    PigServer pigServer;
    try {

        Properties props = new Properties();
        props.setProperty("fs.default.name", "hdfs://localhost:8022");
        props.setProperty("mapred.job.tracker", "localhost:8021");

        System.setProperty("javax.xml.parsers.DocumentBuilderFactory", "com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl");

        pigServer = new PigServer(ExecType.MAPREDUCE, props);

        pigServer.registerScript("/home/training/Desktop/text_v6.pig");



    } 
    catch (ExecException e) {   e.printStackTrace(); } 
    catch (IOException e) { e.printStackTrace(); }

}

The program works perfectly and the output is this:

(2,4L)
(3,2L)
(1,1L)
...

I can potentially have millions of lines like this as an output. That's why I don't want to store the Pig's output into a file, which will be read with Java. I would like to store immediatly the results into a Java structure, like an HashMap for instance.

Is it possible ?

Thanks.

Answer 1

You can retrieve an Iterator over all tuples in the specified alias using the openIterator method in PigServer . This is assuming you have no other dump or store statements in your pig script.

See also: http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html#openIterator(java.lang.String)