Neo4j 3.0.5 backup to Azure failing

Question

When running a backup on Neo4j ent 3.0.5 it fails with the following:

2017-11-03 14:52:43.512+0000 INFO [o.n.c.s.StoreCopyClient] Done, copied 114 
files
2017-11-03 14:52:43.923+0000 INFO [o.n.b.BackupService] Start recovering 
store
command failed: Backup failed: Unexpected Exception

When I check /var/log/neo4j/debug.log it seems to think it's all OK.

2017-11-03 14:52:07.423+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Starting check pointing...
2017-11-03 14:52:07.423+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Starting store flush...
2017-11-03 14:52:08.260+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Store flush completed
2017-11-03 14:52:08.260+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Starting appending check point 
entry into the tx log...
2017-11-03 14:52:08.303+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Appending check point entry 
into the tx log completed
2017-11-03 14:52:08.303+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check 
Pointing triggered by full backup [2012424]:  Check pointing completed
2017-11-03 14:52:11.582+0000 INFO [o.n.b.BackupImpl] BackupServer:6362-6: 
Full backup finished.

I'm running the following commands:

sudo mkdir -m +x -p /backuptoazure
sudo mount -t cifs //xxxx.file.core.windows.net/neo4jbackup /backuptoazure -o vers=3.0,username=xxxx,password=xxxx,dir_mode=0777,file_mode=0777,sec=ntlmssp
sudo neo4j-admin backup --backup-dir=/backuptoazure --name="graph.db-backup.$(date +%Y-%m-%d)"

/backuptoazure is a mountpoint located on Azure files

Points to note;

-When I execute the backup command the files are written to Azure into a temp-copy file (ie it has access) but the files are removed after the failure

-If I run the backup command to a local FS (/tmp) it completes without issue

-I can CP a backup to the mountpoint proving it has access

Any suggestions on where to go next would be appreciated. In particular what is Neo4j when it Start recovering store would be interesting to know

Answer 1

After some digging it seems that during the backup process neo4j starts an instance of the backed up database on the CIFS mount point and in doing so it is trying to use a 'byte range lock' which my assumption is that it is not supported on Azure Files.

Adding the option 'nobrl' to the mount options does the following:

Do not send byte range lock requests to the server. This is necessary for certain applications that break with cifs style mandatory byte range locks (and most cifs servers do not yet support requesting advisory byte range locks).

So the final mount command should be:

sudo mount -t cifs //xxxx.file.core.windows.net/neo4jbackup /backuptoazure -o vers=3.0,username=xxxx,password=xxxx,dir_mode=0777,file_mode=0777,sec=ntlmssp,nobrl

This should work. While not listed explicitly, for more information on the restrictions of Azure Files, please see:

https://docs.microsoft.com/en-us/rest/api/storageservices/features-not-supported-by-the-azure-file-service