Pig：用Java将字符串重新解析为元组

Question

I'll have a Pig script that ends with storing it's contents in a text file.

STORE foo into 'outputLocation';

During a completely different job I want to read lines of this file, and parse them back into Tuples. The data in foo might contains chararrays with characters used when you save Pig Bags/tuples like { } ( ) , etc. I can read the previously saved file using code like.

FileSystem fs = FileSystem.get(UDFContext.getUDFContext().getJobConf());
FileStatus[] fileStatuses = fs.listStatus(new Path("outputLocation"));

for (FileStatus fileStatus : fileStatuses) {
    if (fileStatus.getPath().getName().contains("part")) {
        DataInputStream in = fs.open(fileStatus.getPath());             
        String line;
        while ((line = in.readLine()) != null) {
           // Do stuff
        }
    }
}

Now where // Do stuff is, I'd like to parse my String back into a Tuple. Is this possible/ does Pig provide an API? The closest I could find is the StorageUtil class textToTuple function, but that just makes a Tuple containing one DataByteArray. I want a tuple containing other bags, tuples, chararrays like it originally had so I can refetch the original fields easily. I can change the StoreFunc I save the original file in, if that helps.

Answer 1

This is the plain Pig solution without using JSON or UDF. I have found it the hard way.

import org.apache.pig.ResourceSchema.ResourceFieldSchema;
import org.apache.pig.builtin.Utf8StorageConverter;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.Tuple;
import org.apache.pig.newplan.logical.relational.LogicalSchema;
import org.apache.pig.impl.util.Utils;

Let's say your string to be parsed is this:

String tupleString = "(quick,123,{(brown,1.0),(fox,2.5)})";

First, parse your schema string. Note that you have an enclosing tuple.

LogicalSchema schema = Utils.parseSchema("a0:(a1:chararray, a2:long, a3:{(a4:chararray, a5:double)})");

Then parse your tuple with your schema.

Utf8StorageConverter converter = new Utf8StorageConverter();
ResourceFieldSchema fieldSchema = new ResourceFieldSchema(schema.getField("a0"));
Tuple tuple = converter.bytesToTuple(tupleString.getBytes("UTF-8"), fieldSchema);

Voila! Check your data.

assertEquals((String) tuple.get(0), "quick");
assertEquals(((DataBag) tuple.get(2)).size(), 2L);

Answer 2

I would just output the data into JSON format. Pig has native support for parsing JSON until tuples. It would avoid you having to write a UDF.