Making Array Of Requests Using Requests Library In Python

Question

I am using the requests library to map an array of requests after I get an array from another API request. I am using a loop for the requests but I sure there is a better way to do this because this API request can have 500+ items, so finishing this looping is taking 20+ minutes sometimes.

I tried to use the grequests library and I kept getting recursion complaints. I would love to use an async/map method if possible but after researching apparently the async library is not supported anymore.

self.set_header("Access-Control-Allow-Origin", "*")
response = requests.get("https://hacker-news.firebaseio.com/v0/paststories.json?print=pretty")
data = response.json()
story_list = []

for story in data:
    temp_string = "https://hacker-news.firebaseio.com/v0/item/{}.json?print=pretty".format(story)
    story_data = requests.get(temp_string)
    story_list.append(story_data.json())

There should be a better way to do this looping than the current method because 20+ minutes to get the data is not acceptable. The API response in the original array can return an array of 500+ so the method should be scalable.

Answer 1

requests is synchronous, so your script waits for a response to make a new request. So Maybe you should look into aiohttp and aysnchronous requests.

This could be an example : Is that benchmark reliable - aiohttp vs requests

Answer 2

well, you just need to boost the speed of your io-bond code, there are tones of the solution, please refer to this related answer in stackoverflow: How could I use requests in asyncio?

due to asyncio is too much fundamental, there are a lot of package built base on it, try this package: aiohttp-requests

hope these information can help.