FileNotFoundException - no space left on device

Question

On our production Linux box, we're getting a FileNotFoundException which an additional message of "no space left on device".

I read here and there on the web, that the problem could be related to the count of inodes but seems that's not the problem here. Our system admins are confirming that we have enough inodes .

What else could it be?! I've been digging into this for 2 days now.

I can provide any additional information if needed. Thanks in advance.

Here is the exact stack trace.

13 Nov 2014 10:09:32 ERROR error:113 - Failed to move file: /wr/data/import/xml/YC00210097_ba7b688c-6b46-11e4-b8e8-07cc96f6b24f_1415891227642.xml.
java.io.FileNotFoundException: /wr/data/import/processeddir/YC00210097_ba7b688c-6b46-11e4-b8e8-07cc96f6b24f_1415891227642_2014_11_13_10_09_32_241.xml (No space left on device)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
    at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:671)
    at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:653)
    at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:607)
    at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:1818)
    at com.yellowbook.util.YBFileIterator.finished(YBFileIterator.java:105)
    at com.yellowbook.diadexchange.exchangejob.ImportJob.execute(ImportJob.java:279)
    at com.yellowbook.scheduling.diadexchange.JobExchangeScheduler.executeImportJob(JobExchangeScheduler.java:34)
    at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
    at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:311)
    at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)

Answer 1

The count of the files in processeddir was the problem. So we were getting this (seemingly confusing) FileNotFoundException because there were too many files in the target directory. After we archived a big subset of these files into a compressed (but single) file, the exception disappeared.