Java 多线程网络爬虫，每秒连续提取数据，同时允许消费者检索数据

Question

I am working on a multi-threaded based Web application using Java.

I have two threads inside that application, a web Scraper and a thread to perform some computation( similar to a producer and consumer). The Scraper continuously reads the data from a third party API(world population that updates at each second). The other thread (consumer) continuously tries to retrieve data from scraper and computes the fastest changing rate in every minute.

My question is that the scraper needs to extract data at every second continuously. When the consumer retrieves data, it needs to lock the scraper 's variable(eg.buffer) where data is recorded. However, that may prevent the scraper from recording data at each second continuously. Is there a method that allows the consumer to retrieve data without preventing the scraper from extracting data at every continuous second?

Answer 1

Have a look at the BlockingQueue Java doc. Implementations are threadsafe so your producer and consumer thread can safely communicate with eacher over the queue. If you are worried about "missing a beat" when handing over the scraper result to the consumer, then start a new scraper thread every second. Then, if one scraper thread has to wait when handing over a result, it does not affect the scraping of other threads. If the scraping result structure is timestamped you can deal with possible out of order messages on the consumer level. Or on the queue level by using a PriorityQueue. The PriorityQueue is not threadsafe though.