Too many open files error in Jboss 7

Question

In our production environment currently we are facing the issue like 'Too many files open'. The application is running 2 years without any major problem but all of sudden we got the below error

[org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 2) WFLYDS0012: Scan of /jboss/jboss-eap-7.0/mpayservices/deployments threw Exception: java.lang.RuntimeException: WFLYDS0032: Failed to list files in directory /jboss/jboss-eap-7.0/example/deployments. Check that the contents of the directory are readable.
    at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.listDirectoryChildren(FileSystemDeploymentService.java:1287)
    at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scanDirectory(FileSystemDeploymentService.java:781)
    at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:570)

 java.io.FileNotFoundException: /rsa/rsa_a.pem (Too many open files)
2018-07-11 10:39:13,015 ERROR [stderr] (default task-103)   at java.io.FileInputStream.open0(Native Method)
2018-07-11 10:39:13,015 ERROR [stderr] (default task-103)   at java.io.FileInputStream.open(FileInputStream.java:195)

Here is the code for opening the file

private PrivateKey getPemPrivateKey(String filename) {

        File file = new File(filename);

        try (FileInputStream fileInputStream = new FileInputStream(file);
                DataInputStream dataInputInstream = new DataInputStream(fileInputStream);) {
            byte[] keyBytes = new byte[(int) file.length()];
            dataInputInstream.readFully(keyBytes);
            String privKeyPEM = new String(keyBytes);
            privKeyPEM = privKeyPEM.replace(PRIVATE_KEY_HEADER, "");
            privKeyPEM = privKeyPEM.replace(PRIVATE_KEY_FOOTER, "");

            byte[] decoded = Base64.decodeBase64(privKeyPEM);

            PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(decoded);
            KeyFactory keyFactory = KeyFactory.getInstance(RSA_KEY_FACTORY_INSTANCE);
            PrivateKey key = keyFactory.generatePrivate(spec);
            return key;
        }
        catch (NoSuchAlgorithmException | InvalidKeySpecException | IOException e) {
            e.printStackTrace();
            return null;
        }finally {

        }
    }

We are put the streams in try to block to close the resources automatically as per java 1.7.

But still we are getting the error.But once the server restart it working fine for two three hours after that the issue coming again.

we checked that ulimit -n

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 94482
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

We are checking two days for fixing this issue.But still we cannot get any ideas to close it.

Any help will be greatly appreciated!!!!

Answer 1

As per the above logs, The FileSystemDeploymentService.listDirectoryChildren() is the cause of this above error of deployment.scanner service.

If you're not using the deployment scanner you can simply disable it. It looks though this is likely an OS issue. Make sure the user you're using has permissions to the directory.

JBoss developer document for more references.