使用 python 为同一 url 发送多个 get 请求

Question

I have to send a couple of thousand get requests for the same url. It's quite long when I do that by using for loop, so I looking a better/faster/ ̶s̶t̶r̶o̶n̶g̶e̶r̶- solution. Do you have any ideas?

url = 'https://site.ru:port/api/v2/getRequest'

for index, row in tqdm.tqdm(data.iterrows(), total=data.shape[0]):

        params = {

            'param1_key' : row['param1_value'],
            'param2_key' : row['param2_value'}

        response = requests.get(api_endpoint, params, headers={'apikey': api_key}, timeout=30)

Answer 1

When you do it in a for loop, you have to wait for a response each time. So to make it better/ faster /stronger ... Do it async with requests-futures or grequests !

(Also see Asynchronous Requests with Python requests - it's pretty dated though and points you to grequests anyway)

Answer 2

Hard to say without more info. iterrows if knows not to be fast, but I would bet a coin that the most time consuming part is waiting for the response from the server.

In that case multiprocessing.dummy.Pool can be a handy tool to start multiple concurrent requests without waiting for each to terminate before starting the next one. But beware, sending too many requests to a single server can be seen as an attack...