在hadoop mapreduce作业中插入连接到数据库时发生异常

Question

I have a mapreduce job which is suppose to persist data to the MySQL database but it is throwing an exception which is following,

 INFO mapred.JobClient: Task Id : attempt_201501122044_1310_r_000001_0, Status : FAILED
 java.io.IOException  at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:180)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:597)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)

This mapreduce job runs over the cluster of 5 nodes and I have confirmed that database connection is getting formed by testing simple main class with sql connection creation.

I am badly stuck at this point.

Answer 1

This is a permissions issue: notice the UserGroupInformation is flagged:

    `at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:164`2)

What kind of security issue? It is not clear from the Stacktrace. Is your code doing any HDFS? Or what else that would trigger a security check?