Python multiprocessing dies in the middle

Question

I run a python program on CentOS VPS Server . I am running this program for last few months & I am facing this problem from the beginning.

I usually run this python program from crontab. The program does the following things everyday sequentially.

1. Connect to a MySQL Database & Update some specific info.
2. Divide into 8-10 python instances using multiprocessing .
3. Each instance run virtual display using Xvfb (which is required to run firefox).
4. Each instance run individual firefox browser.
5. Every instance visit one website to check update (using selenium ) for some data and if it get any changes it update to database.
6. After checking each instance close firefox

All these processes usually take 60 min to 90 min if it runs properly. But problem is that it is not running fine every time. Sometime it runs sometimes it dies in the middle. Also I never found any specific reason, why these program dies and what should I do. When I checked top or ps , I found python program is absent and firefox & Xvfb is still there waiting for the python command like a fool.

Probably killed by system due to overload or some other reason. Sometime rebooting system helps. But, I know that is not a good solution. That is why I tried to run the program reducing the niceness value at the lowest level. But, program probably cannot run with negative niceness value, so still the same problem persist.

F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S   500 12673 12655  0  80   0 - 54957 wait   ?        00:00:00 python
1 S   500 12683 12673  0  80   0 - 62766 sk_wai ?        00:00:18 python
1 S   500 12704 12673  0  80   0 - 59295 sk_wai ?        00:00:17 python
1 S   500 12816 12673  0  80   0 - 59230 sk_wai ?        00:00:16 python
1 S   500 12898 12673  0  80   0 - 59400 sk_wai ?        00:00:16 python
1 S   500 12988 12673  0  80   0 - 59491 sk_wai ?        00:00:17 python
1 S   500 13072 12673  0  80   0 - 59151 sk_wai ?        00:00:17 python

Again, I have run this program on my Windows 8 desktop and it runs fine with up to 26 instances (I never tried more :) ). As it is running so many instances at a time page loading become slow. That is why I tried to run this program on 3 VPS server by splitting the job.

Also, I run another program using python , firefox , selenium , Xvfb with single process and it runs fine 10-12 hours without any hiccup. Also this program does almost same thing like first one but only difference is it is running only single process. And, it never create any trouble like first one.

Is there any way, which will confirm that it will run the program properly every time?

Answer 1

I have also faced the same problem. After doing a lot of research I concluded, Firefox is not good option for multiprocessing. Better use PhantomJS or Chrome Driver . In my case when I was saw ps or top, I was getting Zombie Firefox process.