Using JNI in flink YARN cluster jobs

Question

I have an application that dispatches Apache Flink jobs to an AWS Elastic MapReduce YARN cluster through the RemoteExecutionEnvironment scala API.

These jobs use the JNI to run part of their calculations through a C library. During development I just put a System.loadLibrary() call in a RichCrossFunction 's open() method to load this JNI library. This worked fine in a LocalExecutionEnvironment .

Now that I'm moving to a RemoteExecutionEnvironment this no longer seems to work. It looks like Flink is using a new ClassLoader every time it dispatches a job and I am getting Native library already loaded in another classloader errors on the calculation nodes.

Some googling informed me that this is a common issue with Tomcat applications and a solution is available in the Tomcat FAQ: http://wiki.apache.org/tomcat/HowTo#I.27m_encountering_classloader_problems_when_using_JNI_under_Tomcat

Is there a similar solution available for Flink or YARN?

Additionally, is it possible to avoid resubmitting the JAR every time a job is queued? I'm always using the same jar on this cluster, so it's unnecessary overhead...

Answer 1

I fixed the issue by calling loadLibrary in a static initializer in my JNI jar, then dropping my JNI jar in Flink's /lib folder, similar to the pattern in the Tomcat link above.

It is automatically replicated to the Flink TaskManagers by the yarn-session.sh startup procedure. This allowed me to circumvent the ClassLoader segregation in the same way you would do with Tomcat.

I'm using Maven, so I prevented the JNI jar from being included in my uberjar using maven-shade-plugin.

I still don't know if this is the best way, since the flink manual discourages using the /lib folder because it doesn't respect their ClassLoader management ( https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/cluster_execution.html ), but that's exactly what I wanted.

Maybe another way would have been to use a NativeLoader pattern and create a separate temp file for every ClassLoader, but that would create a bunch of duplicate native libraries and this method works for me.