[英]Permission Issue : while running spark job from shell action in Oozie
I am trying to run spark job on Yarn cluster in "yarn-client" mode; 我试图以“ yarn-client”模式在Yarn群集上运行spark作业; using Oozie shell action to issue spark-submit command.
使用Oozie shell动作发出spark-submit命令。 Note that oozie job is triggered by logged in user ("my-user"), but i get Execute permission issue.Please refer log below
请注意,oozie作业是由登录用户(“我的用户”)触发的,但是我遇到了执行权限问题。请参考以下日志
For more detailed output, check application tracking page:http://physrv3:8088/proxy/application_1452252389874_0018/Then, click on links to logs of each attempt.
Diagnostics: Permission denied: user=my-user, access=EXECUTE, inode="/user/yarn/.sparkStaging/application_1452252389874_0018":yarn:hdfs:drwx------
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:180)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:137)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6609)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4223)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:894)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:526)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:822)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
I would suggest you to use a java action as described Spark with Java Action if you have Oozie version smaller than 4.2. 如果您的Oozie版本小于4.2,我建议您使用Java动作描述的Spark使用Java动作 。
Otherwise, if you have admin rights to the oozie server, upgrade to Oozie version >= 4.2.x where there is a native support for Spark actions. 否则,如果您具有oozie服务器的管理员权限,请升级到Oozie版本> = 4.2.x,其中对Spark操作具有本机支持。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.