Spark Scala Unit test getting failed

Question

I am getting this below error while running the test for maven. I have HADOOP_HOME, hadoop.dll file , everything is set up in my local machine and path , environmental variables for my machine. Previously it was running fine. While I had to clone a different repository I started getting this error. Can anyone help me with this.

  org.apache.spark.sql.AnalysisException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V;
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
  at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:196)
  at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
  at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
  at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
  at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
  ...
  Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
  at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
  at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
  at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
  at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:694)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:613)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)
  at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:180)

Answer 1

The problem is that you haven´t set the library path . Try to set the java.library.path java property with the folder where the hadoop.dll is located. If you execute your tests with maven you can set the property of the forked jvm that executes the tests with the argLine option:

mvn -DargLine="-Djava.library.path=[hadoop_dll_dir_path]"