托管节点 web 爬虫？

Question

I've got a crawler which checks a list of URLs every 60s written in Nodejs. It uses no database, stores a few items in-memory and should run 24/7.

What's a proper solution for hosting this crawler?

As far as I've understood AWS it's paid per second which would make a 24/7 process pretty expensive I guess? Or maybe I'm missing something here, the AWS docs are pretty confusing imo.

Answer 1

The tool sounds light enough (based on the purpose), so I'd go for a serverless solution to reduce operations footprint: so go for a Lambda function or ECS FarGate. Here's what you'd expect to pay:

For Lambda, assuming 512MB running for 5 seconds @ 0.0000008333 per 100ms: 60 calls * 24 hours * 30 days -> 43200 * 0.0000008333 50* ~= $1.8 per month

https://s3.amazonaws.com/lambda-tools/pricing-calculator.html

For Fargate at the smallest footprint: 0.25 VCPU and 0.5GB memory: ((0.25 * 0.01239249) + (0.5 * 0.00136079)) * 24 * 30 ~= $2.7 per month

Use caution with those numbers, just a quick draft. Both options are fairly cheap, but Lambda might be easier to work with plus you indicated that you don't need the items in memory to persist calls.