[英]Hbase Kerberos connection from Spark scala
我正在嘗試從spark scala外殼連接到以kerberized保護的Hbase集群,下面是我的代碼,感謝您提供有關錯誤的幫助。 我正在使用以下文件在Spark Shell中傳遞hdfs-site.xml,hbase-site.xml,core-site.xml和我的keytab
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.Connection
import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.TableName
import org.apache.hadoop.security.UserGroupInformation
val conf: Configuration = HBaseConfiguration.create()
conf.set("hbase.zookeeper.quorum", "xxxxx1@abc.com,xxxxx2@abc.com,xxxxx3@abc.com")
conf.set("zookeeper.znode.parent", "/hbase-secure")
conf.setInt("hbase.client.scanner.caching", 10000)
conf.set("hbase.rpc.controllerfactory.class","org.apache.hadoop.hbase.ipc.RpcControllerFactory")
conf.set("hbase.rpc.controllerfactory.class","org.apache.hadoop.hbase.ipc.RpcControllerFactory")
conf.set("hadoop.security.authentication", "kerberos")
conf.set("hbase.security.authentication", "kerberos")
val userGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI("XXX@abc.COM", "/u/xxxxx/XXXX.keytab")
UserGroupInformation.setLoginUser(userGroupInformation)
val connection: Connection = ConnectionFactory.createConnection(conf)
print(connection)
val admin = connection.getAdmin
val listtables = admin.listNamespaceDescriptors()
]
I see a lot of warning in the process as below
warning: Class
找不到org.apache.hadoop.hbase.classification.InterfaceAudience-繼續存根。
錯誤--
WARN AbstractRpcClient: Couldn't setup connection for XXXX@abc.COM to null
RpcRetryingCaller{globalStartTime=1541788150382, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for XXXX@abc.COM to null
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:158)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4427)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4416)
at org.apache.hadoop.hbase.client.HBaseAdmin.listNamespaceDescriptors(HBaseAdmin.java:3123)
... 49 elided
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for XXXX@abc.COM to null
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1560)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1580)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1731)
at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
... 52 more
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for XXXX@abc.COM to null
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:228)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:292)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:62896)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1591)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1529)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1551)
... 56 more
Caused by: java.io.IOException: Couldn't setup connection for XXXX@abc.COM to null
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:665)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:889)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1201)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:218)
... 61 more
Caused by: java.io.IOException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.<init>(HBaseSaslRpcClient.java:117)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:609)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
... 65 more
您的Zookeeper配置缺少Zookeeper端口。 加,
conf.set("hbase.zookeeper.property.clientPort", "2181")
我想建議您一些其他的事情,
之后,您可能會遇到火花配置問題。 復制hdfs-site.xml,hbase-site.xml,core-site.xml和yarn-site.xml(如果使用的是yarn)以生成conf文件夾。
向hbase配置對象添加資源
conf.addResource("/path/to/hbase-site.xml");
設置Java屬性
//Point to the krb5.conf file. Enable Kerberos debug. System.setProperty("java.security.krb5.conf", "/etc/krb5.conf"); System.setProperty("sun.security.krb5.debug", "true");
希望這會有所幫助...
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.