Jmeter - Hive 給 java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer

Question

我正在使用 Jmeter 在 Hive 上創建負載。

在運行 Jmeter 時，我在文件jmeter.log中收到以下錯誤：

 2021-02-05 10:11:02,927 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[HiveThreadGroupTWANG 1-1,5,main]
 java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge23.<clinit>(HadoopThriftAuthBridge23.java:66) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.getBridge(HadoopThriftAuthBridge.java:82) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]

at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.SaslRpcServer
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
    ... 21 more

The extra jars that I have added in the lib/ext directory are: old-hive-exec-3.1.0.3.1.5.0-152.jar hadoop-common-3.1.1.3.1.5.0-152.jar.bkp slf4j-log4j12-1.7.25.jar slf4j-api-1.7.25.jar old-hive-service-3.1.0.3.1.5.0-152.jar old-hive-metastore-3.1.0.3.1.5.0-152.jar old-hive-jdbc-handler-3.1.0.3.1.5.0-152.jar old-hive-jdbc-3.1.0.3.1.5.0-152-standalone.jar.bkp old-hive-jdbc-3.1.0.3.1.5.0-152.jar log4j-1.2.17.jar libthrift-0.9.3-1.jar libfb303-0.9.3.jar hive-service-3.1.2.jar hive-metastore-3.1.2.jar hive-jdbc-3.1.2.jar hive-jdbc-3.1.2.jar hive-exec-3.1.2.jar curator-client-5.1.0.jar curator-framework-5.1.0.jar zookeeper-3.6.2.jar zookeeper-3.3.3.jar hadoop-common-3.1.1.3.1.0.0-78.jar.bkp

將hadoop-common-3.1.1.3.1.5.0-152.jar移動到 lib 目錄后，出現以下錯誤。

  java.lang.NoClassDefFoundError: org/apache/commons/configuration2/Configuration
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:43) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:41) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:143) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:276) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]

      at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.Configuration
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
    ... 24 more

Answer 1

1. 您需要將hadoop-common .jar 文件添加到JMeter 類路徑
2. 依賴庫應該 go 用於lib文件夾，而不是 lib/ext 文件夾，lib/ext 用於JMeter
3. 你有hadoop-common-3.1.1.3.1.5.0-152.jar.bkp ，我不知道什么是.bkp 文件，但我相信它應該有.jar 擴展，很可能你下載了別的東西
4. 你知道jmeter-hadoop插件嗎？ 看起來它對您的用例很有用，至少您將獲得依賴 jars 管理。 有關更多詳細信息，請參閱如何使用 JMeter 文章加載測試 HBase