Migrate SQL Server data to Neo4j and keep data in sync for both the databases

Question

How to migrate data from SQL Server to Neo4j without using LOAD CSV ?

Other tools do not give a complete transformation as per schema but just a sample (ex: ETL). If there is something that could be done with a node js application that would be great.

Also how to keep the data in sync between SQL Server and Neo4j?

Answer 1

You have a few options here: apoc.load.jdbc is one option, there is also neo4j-etl-components project.

apoc.load.jdbc

There is capability for connecting to a relational database and streaming the results of SQL queries to Neo4j from within Cypher available in the APOC procedure library .

For example:

// Create Product nodes
CALL apoc.load.jdbc("jdbc:mysql://localhost:3306/northwind?user=root","SELECT * FROM products") YIELD row 
CREATE (p:Product {ProductID: row.ProductID})
SET p.ProductName = row.ProductName,
    p.CategoryID  = row.CategoryID,
    p.SupplierID  = row.SupplierID

More info is available in the docs.

neo4j-etl-components

neo4j-etl-components is a tool that will

1. Inspect an existing relational database schema
2. Apply rules to this schema to convert to a property graph model
3. Optionally, provides a GUI interface for editing how this translation should be applied
4. Stream export from the relational database to create a new Neo4j database (offline batch) or online incremental insert into Neo4j.

neo4j-etl-components can be used as a command line tool. For example, to run an initial batch import:

~: $NEO4J_HOME/bin/neo4j-etl export  \
  --rdbms:url jdbc:oracle:thin:@localhost:49161:XE \
  --rdbms:user northwind --rdbms :password northwind \
  --rdbms:schema northwind \
  --using bulk:neo4j-import \
  --import-tool $NEO4J_HOME/bin  \
  --csv-directory /tmp/northwind \
  --options-file /tmp/northwind/options.json \
  --quote '"' --force 

See the docs here and Github page for the project here.

Keeping in sync

Once you've done the initial import of course you need to keep the data in sync. One option is to handle this at the application layer, write to a queue with workers that are responsible for writing to both databases, or run incremental versions of your import with apoc.load.jdbc or neo4j-etl-components .