由于Pig MR作业中的权限问题，导致EOFException

Question

I am seeing the following exception in my Map reduce job running on my local hadoop cluster of 80 nodes. This stack trace doesn't point to my code and I suppose it might be related to some permission issue. One interesting thing is that the job is failing like this if I run it as system account. If I run it as myself the job runs just fine.

java.lang.RuntimeException: java.io.EOFException
    at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:103)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:120)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readByte(DataInputStream.java:250)
    at org.apache.pig.impl.io.PigNullableWritable.readFields(PigNullableWritable.java:115)
    at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:100)
    ... 10 more

Any help here is appreciated. Let me know if you need any more information.

Answer 1

This is weird but the permissions were not the issue here. It was that the hadoop configurations being picked up were different for both users. I fixed that and it seems to be working now. Thanks