rsync hangs after transfer over ssh

Question

I wrote a bash script that backs up files from a webserver (HostGator) to a local file server running FreeBSD.

I use rsync over ssh (from the file server) to connect to the remote server (I already have pre-shared rsa keys setup). When I run the following line to start the sync, the files all seem to come in just fine, but the command never returns and the script just hangs forever:

/usr/local/bin/rsync -az --chown=root:admin --chmod=ugo=rwX --exclude ".inode_lock" --rsh='ssh -p2222' admin@domain.com:/home/admin/ '/mnt/blah/blah/LocalBackup/' >> "./Logs/Backup Log.txt"

After waiting a few minutes, when I hit Ctrl+C to stop the command, I poops out the following error messages:

^CKilled by signal 2.
rsync error: unexplained error (code 255) at rsync.c(636) [generator=3.1.2]
rsync error: received SIGUSR1 (code 19) at main.c(1429) [receiver=3.1.2]

This still happens even if the both sides are already synced and it is just checking for changes.

I'm not sure what do to do troubleshoot the problem. I did try removing the -v switch for rsync as some users reported that caused hangs, but I saw no differences.

EDIT

One more additional note. I ran the script again today to continue to troubleshoot. If I leave the script running without disturbing it after it hangs, eventually I receive the following message:

rsync: connection unexpectedly closed (2984632408 bytes received so far) [receiv    er]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receive    r=3.1.2]
rsync: connection unexpectedly closed (8689703 bytes received so far) [generator    ]
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.2]

and then returns back to the command prompt. I'm think this might be due to a timeout on the remote server's end but not sure. But I'm still not sure why the hang is happening though.

UPDATE

I did an additional test and limited the rsync transfer to a specific test folder with some sample files and subfolders, rather than grabbing the entire home directory. When I did this, it was able to successfully complete the transfeer and exit appropriately. So it appears that there must be somee file or folder somewhere in the home directory of the server that is causing the problem. Are there any specific cases where rsync wouldn't be able to transfer a file? I have seen it throw errors while trying to sync files that are write-locked, with a "Permission Denied" error, but even these files didn't stop it from continuing on. Any thoughts?

As an additional note, the remote server I'm connecting to is on a shared hosting account so I don't have root access. I don't know if this could be causing some problems?

UPDATE 2

So I studied the rsync command and added a couple more commandline parameters --progress and --stats (along with --verbose) so I could better understand where it is dying. What I noticed now is that when running the command, where it was hanging was on a verrry large file that was being downloaded from the server. But now with the --progress being reported (I am having it output directly to the terminal for the moment rather than a file), it seems to be moving along just fine, with no hangups so far.

I am now beginning to suspect that maybe the ssh connection is timing out or something due to inactivity? Especially since in the original situation, nothing gets output from the function for a long time while the large file transfer is happening. Is this a possible scenario? If so, what could I do to hold the connection open? (I'm not sure it's a good idea to print the --progress updates directly to the log file).

Answer 1

OK, I figured it out. Apparently HostGator's Shared servers have an SSH timeout limit of 30-45 minutes set by default. Since running rsync took longer than that limit, it was closing the connection on. I called and spoke to their tech support and they got it increased for my server.