Performance Issues with neo4j

Question

I am transferring my rdb to neo4j 2.0.3 using neography . At the beginning my each transfer query used to take very less time, but now(size of the graphml file is around 1.3gbs) the insertion is taking 3-5xtime and often throws this error for certain queries.

org.neo4j.server.rest.repr.RepresentationExceptionHandlingIterable.exceptionOnHasNext(RepresentationExceptionHandlingIterable.java:50)
org.neo4j.helpers.collection.ExceptionHandlingIterable$1.hasNext(ExceptionHandlingIterable.java:46)
org.neo4j.helpers.collection.IteratorWrapper.hasNext(IteratorWrapper.java:42)
org.neo4j.server.rest.repr.ListRepresentation.serialize(ListRepresentation.java:71)
org.neo4j.server.rest.repr.Serializer.serialize(Serializer.java:75)
org.neo4j.server.rest.repr.MappingSerializer.putList(MappingSerializer.java:61)
org.neo4j.server.rest.repr.CypherResultRepresentation.serialize(CypherResultRepresentation.java:83)
org.neo4j.server.rest.repr.MappingRepresentation.serialize(MappingRepresentation.java:41)
org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:160)
org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112)
org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
java.lang.Thread.run(Thread.java:745)

and

org.neo4j.kernel.guard.Guard$Timeout.check(Guard.java:132) org.neo4j.kernel.guard.Guard.check(Guard.java:43) org.neo4j.kernel.InternalAbstractGraphDatabase$5.getNodeByIdOrNull(InternalAbstractGraphDatabase.java:745) org.neo4j.kernel.impl.core.NodeManager.getNodeById(NodeManager.java:301) org.neo4j.kernel.InternalAbstractGraphDatabase.getNodeById(InternalAbstractGraphDatabase.java:1121) org.neo4j.cypher.internal.spi.v2_0.TransactionBoundQueryContext$NodeOperations.getById(TransactionBoundQueryContext.scala:157) org.neo4j.cypher.internal.spi.v2_0.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:131) org.neo4j.cypher.internal.spi.v2_0.TransactionBoundQueryContext$$anonfun$getNodesByLabel$1.apply(TransactionBoundQueryContext.scala:131) org.neo4j.cypher.internal.helpers.JavaConversionSupport$$anon$2.next(JavaConversionSupport.scala:33) scala.collection.Iterator$$anon$11.next(Iterator.scala:328) scala.collection.Iterator$$anon$13.next(Iterator. scala:372) scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389) scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) scala.collection.Iterator$class.foreach(Iterator.scala:727) scala.collection.AbstractIterator.foreach(Iterator.scala:1157) scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176) scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45) scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) scala.collection.AbstractIterator.to(Iterator.scala:1157) scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:257) scala.collection.AbstractIterator.toList(Iterator.scala:1157)

I am importing data of movies from rdb to neo4j by looping over each row of rdb and creating node for a movie (if not existing) and its related tags (if not existing) and then creating relationships between them, and similar such actions.

How can I make the performance better and resolve this error?

This is my neo4j-wrapper.conf

While searching, I landed on http://docs.neo4j.org/chunked/stable/configuration.html which gives a detail performance and configuration tips. But it seems complex to begin with. Are there any to do's and don't's to use neo4j with ruby for better performance .

Please help me out.

Answer 1

This gives a nice background of scaling up...

http://maxdemarzi.com/2013/11/25/scaling-up/#more-2263