JNI，如何在C运行时在JVM之外创建大量对象。

Question

I am currently developing a Java/JNI library to be able to use a C library within Java.

In a sight,

• I pass a file (path/filename) from Java to C,
• the C library opens and reads it,
• the C library performs a set of memory allocations in order to work with the content of the file,
• the C library is then requestable based on its (kind of) dictionary.

My issue is the following:

The loaded file is about 200mo (2.000.000 lines), and its representation in the C library takes more or less the same room. All the content of the file must be stored in RAM as a lot of transformation is done by the library.

But, when the C library malloc objects, after a certain number (overall 5000 lines), I got a problem because of the RAM consumption. It seems, the C library performs malloc using the JVM (allocated) memory.

When I run the same execution using a C runnable (and then calling the C library), all is ok, the 2.000.000 of lines are loaded and the library works properly.

So, it is neither a problem of communication between Java and C nor a problem of passing large objects between the two language, it is a problem of large amount of memory allocations in C when using JNI framework.

Any suggestion? BRgds, Eddy.

Answer 1

So far I know, The objects initialized from C are not initialized in the JVM heap. They are initialized in the C native heap.

As you asked for suggestions, I think loading 2 million lines in RAM is a huge task to do. You may want to split the file into smaller files and process them one by one.

Answer 2

I suggest to create executable wrapper for your C library. In this case standalone executable application will run in another process with own memory space (size depends on operating system).
Next, you can run this executable using ProcessBuilder :

 ProcessBuilder pb = new ProcessBuilder(
                                path to you executable, 
                                path to processed file, 
                                path to out file);
 Map<String, String> env = pb.environment();
 env.put("VAR1", "myValue");
 env.remove("OTHERVAR");
 env.put("VAR2", env.get("VAR1") + "suffix");
 pb.directory(new File("myDir"));
 Process p = pb.start();

In exemple above executable process will write (return) result via out file . Also, you can pass socket address , http url , etc. as command argument to set destination where your java application will obtain result.