Spark Scala 單元測試失敗

Question

我在運行 maven 測試時遇到以下錯誤。 我有 HADOOP_HOME、hadoop.dll 文件，一切都在我的本地機器和路徑中設置，我的機器的環境變量。 以前它運行良好。 當我不得不克隆一個不同的存儲庫時，我開始收到此錯誤。 誰能幫我這個。

  org.apache.spark.sql.AnalysisException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V;
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
  at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:196)
  at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
  at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
  at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
  ...
  Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
  at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
  at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
  at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
  at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:694)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:613)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)
  at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:180)

Answer 1

問題是你沒有設置庫路徑。 嘗試使用hadoop.dll所在的文件夾設置 java.library.path java 屬性。 如果使用 maven 執行測試，則可以使用argLine選項設置執行測試的分叉 jvm 的屬性：

mvn -DargLine="-Djava.library.path=[hadoop_dll_dir_path]"