Apache Pig no output in AWS EMR Hadoop cluster

Question

I'm new to AWS Hadoop cluster.

I deployed a Hadoop cluster via EMR with pig 0.14.0.

I invoked a UDF that has some output to standard output( System.out.println() ).

However, after the pipeline has successfully finished, there's no output from my UDF.

I've also tried stderr and file stream .

But still no output.

Is there anyone encountered a same problem?

How to deal with that?

Answer 1

Your UDF is run on individual nodes of the cluster and the UDF System.out calls will appear in stdout log of the node. They will not be printed to you Pig log.

You can access these logs from Resource Manager which you can easiest find via AWS EMR Console. You can find instructions there how enable access.