从Heroku上的URL检索HTML

Question

I use Heroku to host my telegram bot. The purpose of the bot is to retrieve html from a webpage and convert it to pdf.

After successfully hosting it online, i tried sending an URL to the bot, but it freezes/stops at the moment of sending a get request.

Code:

logger.info('retrieving HTML = {}'.format(url))
page_html = requests.get(url)

logger.info('retrieved HTML')
logger.info('started HTML parsing')
soup = BeautifulSoup(page_html.text, 'html.parser')

In the Heroku logs i only see retrieving HTML = <URL> and than the application does not show any sign of activity.

I tried to connect to the dyno (the app itself on heroku) using the heroku console (accessible from the heroku web page), and entered the following code:

import requests
# url of a recepie
url = 'https://pikabu.ru/story/pirog_quotlen__matushkaquot_5332461'
html = requests.get(url)

the execution of this code in the heroku console takes very long time as well and does not finish (no error, no message, i can stop the process with crtl+c, execution never finishes), and i am not sure what could be the problem.

Thank You in advance, any hint or help would be appreciated.

Answer 1

Without seeing info from your logs, or getting a sense of how big the page you want to scrape is, my guess is that you are hitting Heroku's 30 second timeout.

From the Dev Center article on timeouts :

The request must then be processed in the dyno by your application, and a response delivered back to the router, within 30 seconds to avoid the timeout.

I would check your logs ( heroku logs -t -a yourAppName ) while running the script and look for h12 which is the timeout error code. Or if you are using hobby or above dynos, you could check application metrics on the Dashboard.