如何在两个JVM实例之间共享内存？

Question

I build a huge graph in JVM (Scala) which I want to use repeatedly, tweaking algorithms. I'd rather not reload it each time from disk. Is there a way to have it sit in one JVM while connecting from another, where the algorithms are being developed?

Answer 1

Save your graph to disk, then map it into memory with MappedByteBuffer . Both processes should use the same memory, which will be shared with the page cache.

Answer 2

Two JVMs sounds more complicated than it needs to be. Have you considered doing a kind of "hot deploy" setup, where your main program loads up the graph, displays the UI, and then asks for (or automatically looks for) a jar/class file to load that contains your actual algorithm code? That way your algorithm code would be running in the same jvm as your graph, but you wouldn't have to reload the graph just to reload a new algorithm implementation.

UPDATE to address OP's question in comment:

Here's how you could structure your code so that your algorithms would be swappable. It doesn't matter what the various algorithms do, so long as they are operating on the same input data. Just define an interface like the following, and have your graph algorithms implement it.

public interface GraphAlgorithm {
  public void doStuff(Map<whatever> myBigGraph)
}

If your algorithms are displaying results to some kind of widget, you could pass that in as well, or have doStuff() return some kind of results object.

Answer 3

Did you consider OSGi platform? It lives in a single JVM, but will allow you to upgrade bundles with algorithms without platform restart. Thus you may have a long-term running bundle with your huge data structures and short-term algorithm bundles taking access to the data.

Answer 4

Using RMI perhaps? Have one instance working as server and the rest as clients?

I think it would be much more complicated than reloading from disk.

Answer 5

You can certainly create an interface onto it and expose it via (say) RMI .

My initial thoughts on reading your post, however, are

1. just how big is this graph ?
2. is it possible to optimise your loading procedure instead ?

I know LinkedIn have a vast graph of people and connections that is held in memory all the time and that takes several hours to reload. But I figure that's a truly exceptional case.

Answer 6

If is expensive to build your graph maybe you can serialize the object.

ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream out = new ObjectOutputStream(bos);
        out.writeObject(graph);
        out.flush();
        byte b[] = bos.toByteArray();
//you can use FileOutputStream instead of a ByteArrayOutputStream

Then you can build your object from the file

ByteArrayInputStream inputBuffer = new ByteArrayInputStream(b);
        ObjectInputStream inputStream = new ObjectInputStream(inputBuffer);
        try {
            Graph graph = (Graph) inputStream.readObject();

        } finally {
            if (inputStream != null) {
                inputStream.close();
            }
        }

Just replace the ByteArrayInputStream with a FileInputStream

Answer 7

if the problem is just to dynamicly load and run your code without name clashes a custom class loader could be enough. for a new run just cache all class files in a new classloader.

Answer 8

您是否考虑过使用少量样本数据来测试算法？

Answer 9

Terracotta在许多JVM实例之间共享内存，因此您可以轻松地将群集应用于您的系统。

Answer 10

Woo! late to the party.

If its on a local machine, similar to mapped byte buffers, there is apache direct memory. http://directmemory.apache.org/

If you want it distributed, give http://hazelcast.org/ a try. Its used by a lot of large projects. Of course, your objects must be serializable.