Spark YARN应用程序中的Kerberos中的HDFS写问题

Question

I have a spark application, which reads data from Kafka and writes the data in HDFS. My application works fine for few minutes but after some time it starts giving below error and get failed.

2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] UserGroupInformation - PriviledgedActionException as:username@REALM_NAME (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 Spark Context Cleaner [INFO ] ContextCleaner - Cleaned accumulator 3480439
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] Client - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [WARN ] UserGroupInformation - PriviledgedActionException as:username@REALM_NAME (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
2018-01-02 17:59:20 Spark Context Cleaner [INFO ] ContextCleaner - Cleaned accumulator 3480438
2018-01-02 17:59:20 LeaseRenewer:username@nameservicename [INFO ] RetryInvocationHandler - Exception while invoking renewLease of class ClientNamenodeProtocolTranslatorPB over namenode1/10.12.2.2:8020. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]; Host Details : local host is: "edgenode/10.12.2.1"; destination host is: "namenode1":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1508)
    at org.apache.hadoop.ipc.Client.call(Client.java:1441)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy41.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:590)
    at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy42.renewLease(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:945)
    at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
    at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
    at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
    at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Clients credentials have been revoked (18))]
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
    at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    ... 16 more

Let me know if anyone know the resolution for this issue.

Answer 1

The issue got resolved. I was getting this issue because the keytab file was missing on one of the machine.