How to wait for a POST request (requests.post) completion in Python?

Question

I'm using the requests library in Python to do a POST call. My POST call takes about 5 minutes to be completed. It will create a file in a S3 bucket. After that, I want to download this file. However, I need to create an extra logic to wait for my POST to finish before executing the next line on my code to download the file.

Any suggestions?

Is it possible to use the subprocess library for this? If so, how would be the syntax?

Code:

import requets
r = requests.post(url)
# wait for the post call to finish
download_file(file_name)

Answer 1

It should already wait until it's finished.

Python, unlike Node.js, will block requests by default. You'd have to explicitly run it in another thread if you wanted to run it async. If it takes your POST request 5 minutes to fetch, then the download line won't run until the 5 minutes are up and the POST request is completed.

Answer 2

The question says the POST request takes 5 minutes to return, but maybe that's not quite right? Maybe the POST request returns promptly, but the server continues to grind 5 minutes creating the file for the S3 bucket? In that case, the need for a delay makes sense. The fact that a separate download is needed at all tends to support this interpretation (the requested info doesn't come back from the request itself).

If a failed download throws an exception, try this:

import time
r = requests.post(url)
while True:
    time.sleep(60) # sixty second delay
    try:
        download_file(file_name)
        break
    except Error:
        print ("File not ready, trying again in one minute")

Or if download_file simply returns False on failure:

import time
r = requests.post(url)
while True:
    time.sleep(60) # sixty second delay
    if download_file(file_name):
        break
    print ("File not ready, trying again in one minute")

Since my interpretation of the question is speculative, I'll delete this answer if it's not to the point.

Answer 3

Michael's answer is correct. However, in case you're running Selenium to crawl the webpage, the frontend JS takes some time to appropriately render and show the request result. In such scenarios I tend to use:

import time
time.sleep(5)

That said, in such cases you have explicit and implicit waits as other options, too. Take a look at: Slenium Waits documentation .

In case you're directly sending requests to the API, Python waits for the response until it's complete.