Python-提出超过100,000个请求并将其存储到数据库的最佳方法

Question

There is data that I'm scraping from many different requests.

Up until now, I've used multithreading and the requests library to retrieve the necessary and then loading them into an sqlite database. with approximately the following approach:

p = Pool(processes=8)
for x in range(start_1,end_1):
    for y in range(start_2,end_2):
        entry_list = p.starmap(get_data, [(x , y , z) for z in range(start, end)]):
        ### get_data makes the request and retruns a tuple of (x,y,z,data)
        for entry in entry list:
            cur.execute('''INSERT INTO Database (attrib_1, attrib_2, attrib_3, data )
            VALUES ( ?, ?, ?, ?)''', entry )

This approach is very slow (will take days to make all of the requests on my machine). After doing a little research I have seen that there are alternatives to multithreading for this kind of problem, such as asynchronous requests. Unfortunately, I don't know anything about this approach and whether or not it's appropriate, far less how to implement it.

Any advice on how to complete this task efficiently would be greatly appreciated.

Answer 1

Since your program is I/O bound, look at event loops. True multi-threading is broken in Python because of the global interpreter lock (GIL).

Look at asyncio (available since Python 3.4) and/or Twisted .