Azure 中的 Python 网络爬虫

Question

I have few web scrapers written in python to scrape data from ecommerce websites. I want to run these scripts on Azure. What's the best way out? I am looking into Azure functions and Azure Webjobs since I want to have a automated schedules to trigger my scripts. Considering the amount of data to be scraped is huge, will Azure functions will be a costly affair ? And how scalable will be webjobs to get this job done? Are there any other cost effective ways to achieve the task ? (I am New to Azure).

Answer 1

You can try using the Selenium and HTTP Trigger for web scraping .

For Azure function cost on Consumption plan and Premium plan, you can refer to Azure Functions pricing and Reddit/Azure : Azure hosted Web Scraper, good or bad idea?

You can refer to Run Selenium on azure functions , How to create a Selenium web scraper in Azure Functions , Running webscraping Python file with HTML trigger on Azure and Create a Website Scraper for Azure Functions