如何在 AWS EC2 上的 beautifulsoup 进程被杀死之前破坏它？

Question

I want to run a python process that involves web scraping using beautifulsoup 24/7 on some.org websites. It runs smoothly on the majority of websites, however, for some rare exceptions, there is a spike in Network, as you can see from the image below. The python process (not the instance) gets killed.

When scraping I avoid any.pdf or.jpg... so that the CPU usage never exceeds 15% (so far, no issues)

Ideally, I would need a script that makes the process restart when is killed, but I have been told is not possible (unless some of you suggest me otherwise, perhaps an AWS service that I can configure to make a process restart when killed).

Is there a way to limit the beautifoulsoup network capability and stop the scraping of that particular link before it reaches a network peak?

Answer 1

Ideally, I would need a script that makes the process restart when is killed, but I have been told is not possible

Not sure if this is true.

1. The easiest example that proves that you can do that is to create a bash script that checks if your python script/process is running and if not, it starts it. The script can be scheduled using CRON to check every X minutes/seconds whatever you want.

Simple example:

#!/bin/bash
ps -eaf | grep <your-process>
if [ $? -eq 1 ]
then
<start your script again>
else
echo "running"
fi

2. The more involved solution would be to setup your process/script as a systemd service so that it automatically restarts your process when it crashes.

See for example:

• Setup a python script as a service through systemctl/systemd
• Auto-restart a crashed service in systemd