Linking native .so in Google Dataproc Spark

Question

I have a native library (Freeling), which I have compiled using cmake and make natively and installed through cluster startup actions (so, it should exist in master and each worker)

Even so, I'm getting this error when calling System.loadLibrary

Exception in thread "main" java.lang.UnsatisfiedLinkError: no Jfreeling in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)

I've tried using the following properties to get the program to find the library (It's called in a static block of code)

      "properties": {
        "spark.driver.extraClassPath": "/usr/local/share/freeling/APIs/java/Jfreeling.jar:/usr/local/lib/libfreeling.so",
        "spark.executor.extraClassPath": "/usr/local/share/freeling/APIs/java/Jfreeling.jar:/usr/local/lib/libfreeling.so",
        "spark.executor.extraLibraryPath": "/usr/local/lib/libfreeling.so",
        "spark.driver.extraLibraryPath": "/usr/local/lib/libfreeling.so",
        "spark.executorEnv.LD_PRELOAD": "/usr/local/lib/libfreeling.so",
        "spark.yarn.dist.files": "/usr/local/lib/libfreeling.so",
        "spark.yarn.appMasterEnv.LD_PRELOAD": "libfreeling.so",
        "spark.files": "/usr/local/lib/libfreeling.so",
        "spark.executorEnv.LD_LIBRARY_PATH": "libfreeling.so"
      },
      "jarFileUris": [
        "file:///usr/local/share/freeling/APIs/java/Jfreeling.jar",
        "file:///usr/local/lib/libfreeling.so"
      ],

Answer 1

Can you try putting your library under /usr/lib/hadoop/lib/native/ ? In /etc/spark/conf/spark-env.sh , it has

# Spark got rid of SPARK_LIBRARY_PATH in 1.0
# It has properties for extraLibraryPaths, but this is more extensible
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native

Answer 2

您应该将 /usr/local/share/freeling/APIs/java/Jfreeling.jar 添加到您的 CLASSPATH，并将 /usr/local/share/freeling/APIs/java/libJfreeling.so 添加到您的 LD_LIBRARY_PATH。