Concurrent gremlin-server and graph queries in java

Question

I'm writing a Java application with TinkerPop3. It communicates with a Neo4j graph and sends read/write queries through Gremlin using the neo4j-gremlin 3.0.0.M7 libraries.

Concurrently, I would like to make this graph available through HTTP using gremlin-server 3.0.0.M7 . Separately, these operations work perfectly fine. However, it seems that this is not possible concurrently due to multiple connections not being allowed (ie the GremlinServer object and the Java code both try to obtain the lock on the graph).

Of course, a workaround could involve creating a Client from within the Java program and connect it to the server. However, I would rather eliminate the communication overhead this introduces.

The big question: is this possible?

For completeness, here's my minimal code. Note that my gremlin-server-neo4j.yaml refers to the standardly included neo4j-empty.properties file, containing the same neo4j graph data directory as the Neo4jGraph object in my Java code (ie /tmp/neo4j ).

import com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph;
import com.tinkerpop.gremlin.server.GremlinServer;
import com.tinkerpop.gremlin.server.Settings;    

public class Main {

    Neo4jGraph g;
    GremlinServer s;

    public static void main (String[] argv) {
        new Main().start();
    }

    private void start () {

        try {
            Settings settings = Settings.read(getClass().getResourceAsStream("/gremlin-server-neo4j.yaml"));
            s = new GremlinServer(settings);
            s.run();
        } catch (Exception e) {
            e.printStackTrace();
        }

        g = Neo4jGraph.open("/tmp/neo4j");

        // Gremlin code here

        g.close();
        s.stop();
    }
}

And, finally, the exception:

Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /tmp/neo4j
    at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.<init>(Neo4jGraph.java:160)
    at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.open(Neo4jGraph.java:175)
    at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.open(Neo4jGraph.java:184)
    at org.test.Main.start(Main.java:33)
    at org.test.Main.main(Main.java:15)
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /tmp/neo4j
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:366)
    at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory$1.newDatabase(GraphDatabaseFactory.java:91)
    at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:181)
    at com.tinkerpop.gremlin.neo4j.structure.Neo4jGraph.<init>(Neo4jGraph.java:133)
    ... 4 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.StoreLockerLifecycleAdapter@67ec8477' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115)
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:343)
    ... 8 more
Caused by: org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: /tmp/neo4j/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
    at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:82)
    at org.neo4j.kernel.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:44)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507)
    ... 10 more
Caused by: java.io.IOException: Unable to lock org.neo4j.kernel.impl.nioneo.store.StoreFileChannel@baf1bb3
    at org.neo4j.kernel.impl.nioneo.store.FileLock.wrapFileChannelLock(FileLock.java:38)
    at org.neo4j.kernel.impl.nioneo.store.FileLock.getOsSpecificFileLock(FileLock.java:93)
    at org.neo4j.kernel.DefaultFileSystemAbstraction.tryLock(DefaultFileSystemAbstraction.java:93)
    at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:74)
    ... 12 more

Answer 1

As you've found, what you are trying to do won't work because two separate processes cannot work on the embedded Neo4jGraph . As it stands you can't get access to the configured graph instances from the GremlinServer object and I'm not sure I'd change that as I'm not completely sure that this is a feature that would be useful to everyone.

It looks to me like what you need is a way to "initialize" your Graph instance. If so, then Gremlin Server offers a way to do that. You can supply an initialization script in the yaml file (the below is a snippet from conf/gremlin-server-classic.yaml which is packaged with Gremlin Server distribution).

scriptEngines: {
  gremlin-groovy: {
    imports: [java.lang.Math],
    staticImports: [java.lang.Math.PI],
    scripts: [scripts/generate-classic.groovy]}}

Note the "scripts" key which lets you supply script files to execute. Those scripts will yield you access to "g" (or whatever graphs you have configured). For this example, scripts/generate-classic.groovy just has:

TinkerFactory.generateClassic(g)

In this way you can do all your initialization work on the Graph that will ultimately be hosted by Gremlin Server when it starts and can then simply start it all up in the standard way with bin/gremlin-server.sh .