java.lang.UnsatisfiedLinkError caused by “Too many open files” error

Question

I am trying to diagnose a problem in a large 3rd-party application (Midonet) that is happening when JNA tries to load the "c" library. The log entry (with stacktrace) is below.

The problem manifests when I try to start midonet on a new machine with a Linux 5.3 kernel. It is working on an older kernel (4.15). The open file limit is 4096 so I am assuming that it is the library loading that is (somehow) opening bazzilions of files.

Now Midonet is using a rather old version of JNA (4.2.2) so I have bumped to the last of the 4.x versions to see if it would help. The next thing I can try is rebuilding JNA 4.x on the platform itself.

I haven't been able to find any relevant issues in either the JNA or Midonet issue trackers. Or anything relevant with Google searches

I would like to know if anyone else has seen this error before (with Midolnet or any other Java application). Other suggestions about how to debug this would be appreciated.

Here is the stacktrace:

2020.05.22 16:49:40.212 ERROR [agent-services-pool-1] jna -  Native method calls are not available
java.lang.UnsatisfiedLinkError: Failed to create temporary file for /com/sun/jna/linux-x86-64/libjnidispatch.so library: Too many open files
    at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:962) ~[midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at com.sun.jna.Native.<clinit>(Native.java:190) ~[midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.jna.CLibrary.<clinit>(CLibrary.java:45) ~[midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.netlink.UnixDomainChannel.<init>(UnixDomainChannel.java:61) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.netlink.UnixDomainChannelImpl.<init>(UnixDomainChannelImpl.java:32) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_252]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_252]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_252]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_252]
    at org.midonet.netlink.NetlinkSelectorProvider.makeInstanceOf(NetlinkSelectorProvider.java:143) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.netlink.NetlinkSelectorProvider.openUnixDomainSocketChannel(NetlinkSelectorProvider.java:118) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.services.rest_api.hacks.UnixDomainServerSocketChannel$.open(UnixDomainServerSocketChannel.scala:38) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.services.rest_api.hacks.UnixDomainServerSocketChannel.open(UnixDomainServerSocketChannel.scala) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.services.rest_api.hacks.UnixDomainServerConnector.open(UnixDomainServerConnector.java:341) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.services.rest_api.hacks.UnixDomainServerConnector.doStart(UnixDomainServerConnector.java:258) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.eclipse.jetty.server.Server.doStart(Server.java:384) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.services.rest_api.BindingApiService.doStart(BindingApiService.scala:113) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:203) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.minion.Daemon$$anonfun$org$midonet$minion$Daemon$$startMinion$1.apply(Daemon.scala:87) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at org.midonet.minion.Daemon$$anonfun$org$midonet$minion$Daemon$$startMinion$1.apply(Daemon.scala:83) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [midolman.jar:5.4.6-Nectar-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_252]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_252]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_252]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_252]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]

The Midonet code that is triggering the code is:

import com.sun.jna.Native;
import com.sun.jna.Platform;

...

static {
    try {
        Native.register(Platform.C_LIBRARY_NAME);
    } catch (NoClassDefFoundError | UnsatisfiedLinkError | NoSuchMethodError e) {
        log.error("Native method calls are not available", e);
        System.exit(-1);
    }
}

Answer 1

It turns out that the problem wasn't JNA's fault at all.

It turns out that the midolnet minion process really does need > 4096 open files when run on a Linux 5.3 kernel. Setting ulimit -n 5000 in the minion-start script fixes it.