I have few web scrapers written in python to scrape data from ecommerce websites. I want to run these scripts on Azure. What's the best way out? I am looking into Azure functions and Azure Webjobs since I want to have a automated schedules to trigger my scripts. Considering the amount of data to be scraped is huge, will Azure functions will be a costly affair ? And how scalable will be webjobs to get this job done? Are there any other cost effective ways to achieve the task ? (I am New to Azure).
You can try using the Selenium and HTTP Trigger for web scraping .
For Azure function cost on Consumption plan and Premium plan, you can refer to Azure Functions pricing and Reddit/Azure : Azure hosted Web Scraper, good or bad idea?
You can refer to Run Selenium on azure functions , How to create a Selenium web scraper in Azure Functions , Running webscraping Python file with HTML trigger on Azure and Create a Website Scraper for Azure Functions
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.