I keep getting '[Errno 101] Connection timed out' when running python script on Airflow

Question

I keep getting '[Errno 101] Connection timed out' when I tried to call an API using Python on Airflow that provided by Google Cloud Composer.

Here's my code:

r1 = requests.post(
    url1,
    params=request1_params,
    headers=request1_headers
)

a_token = r1.json().get('access_token')

And the response when I tried to trigger DAG is this:

[2021-10-04 01:21:33,056] {taskinstance.py:1152} ERROR - HTTPSConnectionPool(host='xxx.com', port=443): Max retries exceeded with url: /sf/xxx/token?grant_type=password&username=???%40???&password=???%2???&client_id=????&client_secret=???? (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f43a2958860>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/opt/python3.6/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/opt/python3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/opt/python3.6/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out

I've read some articles that you need to include proxy when sending requests, but what proxy should I used?

Any help is appreciated!

edit: I've found out that the problem is on the airflow that I am using. It says that private network need NAT configured. So, after installed few libraries and setup NAT, it now can send requests to API.

Thank you for all the help.

Answer 1

Most probably this has nothing to do with proxy but rather a malformed request. This looks like a part of an OAuth flow, but any particular reason for not using operator? Eg https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#simplehttpoperator , you can then set the the timeout via extra_options as those are from requests lib https://github.com/apache/airflow/blob/main/airflow/providers/http/operators/http.py#L58