使用ASP.NET MVC应用程序定期插入Azure数据库

Question

I am wondering how to implement a solution that will retrieve data that I have scraped, and use it to display in an ASP.NET MVC web application.

The current implementation scrapes the data and displays it from the controller to the view, however by doing so, the request to view the web page will take very long due to the scraper running when a request to view the page with scraped data is processed.

Is there any implementation I can do to separate the data retrieval and the website?

Currently I have a console application scraper class that scrapes data, and a ASP.NET MVC web application that will display the data. How can I couple them together easily?

Answer 1

Based on system size I think you can do 2 things:

• Periodically scrape data and save it in the memory
• Periodically scrape data and save it in the database

It is oblivious that if scrapped data is big you need to store it in database, otherwise you can keep it memory and highly boost performance.

Running tasks in asp.net periodically is covered by background workers. Some easy way to periodically run tasks is to initiate thread in Application_Start . I don't go more deeply in implementation, because it is already answered. You can reed it here: Best way to run scheduled tasks

For saving data in memory you can use something like this:

public static class Global
{
    public static ConcurrentBag<ScrapedItem> ScrapedItems;
} 

*Note, it is necessary to use thread-safe collection, because of getting and adding to this collection will be done from different threads: one from background worker, one from request. Or you can use lock object when getting/setting to not thread safe collection.