Speeding up Neo4j running in Azure

Question

We have a long running program that copies data from a SQL Server database to a Neo4j database. It currently takes over 12 hours. For each record from the SQL Server database, the program has to do dozens of accesses to Neo4j. I cannot change this.

The program and Neo4j run in an Azure worker role. Neo4j runs in its own service.

We tried to speed up the process by upgrading from a Medium machine (2 x 1.6GHz CPU, 3.5GB RAM, 490GB Storage) to a Large machine (4 x 1.6GHz CPU, 7GB RAM, 1,000GB Storage) but this didn't speed up the program.

I cannot change the number of accesses to Neo4j, but I'm hoping there are ways to speed up each access. Any suggestions on speeding up Neo4j accesses running in an Azure worker role would be greatly appreciated.

Answer 1

For write heavy stuff, make sure you have your mmio tweaked so that the store files can fit in RAM if possible. See some tips on that here: http://docs.neo4j.org/chunked/milestone/configuration-io-examples.html

You say you upgraded RAM, but you also need to adjust JVM heap settings: http://docs.neo4j.org/chunked/stable/configuration-jvm.html

Finally, if you're doing a lot of writing and no reading, you might try disabling the cache to avoid the churn of neo's object cache. In mixed read/write scenarios this is a bad idea, but in heavy write scenarios, I've seen a nice throughput increase.

Anyway, please give us more details: code samples, cypher queries, and configuration files--as well as some performance stats.