AWS lambda function in VPC unable to put item in Dynamodb table

Question

This is the error I keep getting

[DEBUG] 2018-11-09T16:56:03.543Z    eaa6c2df-e43f-11e8-ae27-cf0081fe803c    ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 134, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 88, in create_connection
raise err
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/connection.py", line 78, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 204, in connect
conn = self._new_conn()
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connection.py", line 139, in _new_conn
(self.host, self.timeout))
botocore.vendored.requests.packages.urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/botocore/vendored/requests/adapters.py", line 370, in send
timeout=timeout
File "/var/runtime/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/var/runtime/botocore/vendored/requests/packages/urllib3/util/retry.py", line 271, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
botocore.vendored.requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/botocore/endpoint.py", line 222, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/var/runtime/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/var/runtime/botocore/vendored/requests/adapters.py", line 419, in send
raise ConnectTimeout(e, request=request)
botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='dynamodb.eu-west-2.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f77b6b3fd30>, 'Connection to dynamodb.eu-west-2.amazonaws.com timed out. (connect timeout=60)'))

The crazy part is my code worked just fine yesterday. And now all of a sudden my lambda function seems not to be able to access the dynamodb. They are connected via a VPC endpoint and it worked fine yesterday. This has happened to me before and when I just left it for the night it ended up working the next day. This is unacceptable behaviour though and I need to find out what is causing it.

Any ideas?

Answer 1

I couldn't get this to work simply using the new vpc endpoint feature so I went for the old way of doing things which is using a nat gateway. Here's a good link on how to do it: https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7