Python Web 抓取 - 如何 24/7 全天候抓取新闻网站以获取新文章？

Question

Didn't find an answer regarding my particular question, so I'm sorry if this has been asked already.

I've created a Python program that scrapes the articles posted on news websites by certain keywords. On average, when running it once in the evening, it would be searching through 2000 articles of the day. Now I obviously want this program to run on loop 24/7 looking for new articles in realtime (or every 5 minutes). When it hits something based on my keywords, I get notified.

Therefore, I wanted to know whether you guys have any good recommendations on hosting? I've heard about AWS Lambda but wanted to get a second opinion. Anything that costs below -$250 a month is possible:) Maybe someone has a similar project running or can confirm my idea with AWS. Thanks in advance!

Answer 1

There are basically 2 options that come to mind:

1. You can either provide your own host to run your code 24/7, eg an old laptop or PC you're not using, effectively paying only for electricity. This method won't allow for any scaling if you wish to scale up later though (assuming you don't want to buy a new hardware).
2. You can use public cloud (AWS, GCP, etc). AWS Lambda or a dedicated EC2 are the first that come to mind, they are relatively easy to set up and run code on. Actual costs can vary depending on AWS region, instance type, usage time, and other factors (eg will you be using S3 as well?), but you could keep them below $250 per month without too much trouble. Small size Lambdas and EC2s are quite cheap to use and you could easily scale up if you need more resources.

Option 2 is better:)

Answer 2

Great question, once your script starts do you ever run new scripts or can you just leave the terminal running?

In the latter case, you need Amazon ec2, not Lambda. Lambda is for running functions, an Ec2 is the "cloud computer" that you are looking for to "host" and run your program.

Look into Ec2, and use EBS or EFB for storage. S3 is good for storing images, or links, or objects, but if you are using an Ec2 instance (cloud computer) and don't need to store your data as an object and don't need to use a dedicated MYSQL or NOSQL database, just store the info in your EBS or EFB. Remember, EBS and EFB are the hard drive of the computer (your ec2), and Amazon RDS is database, Amazon Aurora is inside RDS and is for MYSQL, PostGRESL, and S3 is like a image / object drive. For example, if you had an ebook you were going to distribute, you would store your ebook in S3.

You can set up an Ec2 and EBS for free too. Just use the free tier and use the t2.micro for ec2 instance. See how it runs for a few days and then go bigger when necessary.