SSH shuts down with broken pipe before python program finishes matplotlib part

Question

I am running a python 3 program via SSH which takes a few hours to finish. After all computations the program should save a matplotlib figure and then stop. What happens is that after all computations a Write failed: broken pipe error is shown and when I log back in the figure is not updated.

Another question mentioned the process spawns child processes for the figure which might be killed when the main program finishes computation, so I ran it with nohup to no avail. What other options can I try out?

Answer 1

If I had a guess here, it is that your SSH TCP connection is getting dropped. On my systems, I set the following SSH client parameters to help in this situation.

ServerAliveInterval = 300
ServerAliveCountMax = 300

This keeps data flowing over your connection even while your terminal is awaiting output from the running process.