How to access a non-Google MySQL server database (no Cloud SQL!) from Google Cloud Function in Python runtime using SQLAlchemy

Question

I try to connect from a Google Cloud Function in Python runtime to an external MySQL server db that is not hosted by Google Cloud.

My "requirements.txt":

# Function dependencies, for example:
# package>=version
SQLAlchemy>=1.4.2
PyMySQL==1.0.2

Core code of the Cloud Function:

from os import environ 
import sqlalchemy

db_user = environ["DB_USER"]
db_pass = environ["DB_PASS"]
db_name = environ["DB_NAME"]
db_host = environ["DB_HOST"]
db_port = environ["DB_PORT"] # if not used, default 3306 anyway

db_address = f"""mysql+pymysql://{db_user}:{db_pass}@{db_host}/{db_name}?charset=utf8&use_unicode=1"""
db_engine = sqlalchemy.create_engine(db_address)

When I test the Cloud Function in the "Testing" tab, pressing on Test the function , I get the error:

 for termination reason. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging Details: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. ```

And in the log:

Or as searchable text:

Debug2022-01-07T14:36:29.221892880ZMYCLOUDFUNCTIONdvaw7xewhjqj Function execution started Default2022-01-07T14:36:29.529ZMYCLOUDFUNCTIONdvaw7xewhjqj OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k Debug2022-01-07T14:36:30.964665907ZMYCLOUDFUNCTIONdvaw7xewhjqj Function execution took 1743 ms, finished with status code: 200 Debug2022-01-07T14:36:50.088620704ZMYCLOUDFUNCTIONdvawxkmbid1w Function execution started Function execution started Default2022-01-07T14:36:50.340ZMYCLOUDFUNCTIONdvawxkmbid1w 2022-01-07 14:36:50,267 [ERROR]: Exception on / [POST] 2022-01-07 14:36:50,267 [ERROR]: Exception on / [POST] Error2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_disp atch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise raise value File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/ init .py", line 99, in vi ew_func return function(request._get_current_object()) File "/workspace/main.py", line 139, in get_csv_in_tmp_and_move_to_gs engine = sqlalchemy.create_engine(db_address) File "", line 2, in create_engine File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/layers/google.python.pip/pip/li b/python3.9/site-packages/flask/_compat.py", line 39, in reraise raise value File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/ init .py", line 99, in view_func return function(request._get_current_object()) File "/workspace/main.py", line 139, in get_csv_in_tmp_and_move_to_gs engine = sqlalchemy.create_engine(db_address) File "", line 2, in create_engine File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned Default2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w r eturn fn(*args, **kwargs) return fn(*args, **kwargs) Default2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 560, in create_engine File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 560, in create_engine Default2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w dbapi = dialect_cls.dbapi(**dbapi_args) dbapi = dialect_cls.dbapi(**dbapi_args) Default2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 163, in dbapi File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 163, in dbapi Default2022-01-07T14:36:50.341ZMYCLOUDFUNCTIONdvawxkmbid1w return import ("MySQLdb") return import ("MySQLdb") Debug2022-01-07T14:36:50.342294068ZMYCLOUDFUNCTIONdvawxkmbid1w Function execution took 254 ms, finished with status: 'crash' Function execution took 254 ms, finished with status: 'crash' ```

When I use a with statement for the connection lifetime, I get a slightly different content, but the problem is the same, it cannot connect to the db:

Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise raise value File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1936, in dispa tch_request return self.view_functionsrule.endpoint File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/ init .py", line 99, in view_func return function(request._get_current_object()) File "/workspace/main.py", line 177, in get_csv_in_tmp_and_move_to_gs connection = engine.connect() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3204, in connect return self._connection_cls(self, close_with_result=close_with_result) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in init else engine.raw_connection() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3283, in raw_connection return self._wrap_pool_connect(self.pool.connect, _connection) File "/layers/google.python.pip/pip/ lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3253, in _wrap_pool_connect Connection. handle_dbapi_exception_noconnection( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2100, in handle_dbapi_exception_noconnection util.raise ( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect return fn() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 310, in connect return _ConnectionFairy._checkout(self) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalch emy/pool/base.py", line 476, in checkout rec = pool._do_get() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 146, in do_get self. dec_overflow() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise ( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get return self._create_connection() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection return _ConnectionRecord(self) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 371, in init self.__connect() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 666, in connect pool.logger.debug("Error on connect(): %s", e) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise ( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect self.dbapi_connection = connection = pool._invoke_creator(self) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 590, in connect return dialect.connect(*cargs, **cparams) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 597, in connect return self.dbapi.connect(*cargs, **cparams) File "/layers/google.python.pip/pip/lib/python3.9/site-package s/pymysql/connections.py", line 353, in init self.connect() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/pymysql/connections.py", line 664, in connect raise exc sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'MY_SERVER_ADDRESS' (timed out)") Default 2022-01-09T23:03:25.322Z MYCLOUDFUNCTIONearoz5s8jozu (Background on this error at: https://sqlalche.me/e/14/e3q8 ) (Background on this error at: https://sqlalche.me/e/14/e3q8 )

I tested the same code and setting on my local machine and in a local Docker container and both can connect with the same connection string (db_url).

Question

How can I get the connection to a non-Google MySQL server?

Some ways to go perhaps

Idea

Is this just a permission issue, for example, should I add some permissions to my user's role?

Idea

Or could it be that Google Cloud Functions do not allow any access to servers outside the Google universe?

Idea

And if only Cloud SQL servers are accepted as sources in Cloud Functions (which might be the case since I found only guides for this case like the official one from Google: Connecting from Cloud Functions to Cloud SQL or this Example of how to use MySQL in a Google Cloud Function ), could I somehow make the external server a Google accepted Cloud SQL server without actually uploading it there, or is there any other workaround?

Idea

The connection example from Cloud Functions to Google Cloud SQL (MySQL) asks you to activate the Cloud SQL API as a first step and then uses sockets in the connection string:

# Remember - storing secrets in plaintext is potentially unsafe. Consider using
# something like https://cloud.google.com/secret-manager/docs/overview to help keep
# secrets secret.
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
instance_connection_name = os.environ["INSTANCE_CONNECTION_NAME"]

pool = sqlalchemy.create_engine(
    # Equivalent URL:
    # mysql+pymysql://<db_user>:<db_pass>@/<db_name>?unix_socket=<socket_path>/<cloud_sql_instance_name>
    sqlalchemy.engine.url.URL.create(
        drivername="mysql+pymysql",
        username=db_user,  # e.g. "my-database-user"
        password=db_pass,  # e.g. "my-database-password"
        database=db_name,  # e.g. "my-database-name"
        query={
            "unix_socket": "{}/{}".format(
                db_socket_dir,  # e.g. "/cloudsql"
                instance_connection_name)  # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
        }
    ),
    **db_config
)

Is it possible that without such API and without sockets to secure the connection, the SQL query cannot be fired to a MySQL server that is not a Google Cloud SQL server?

Idea

Or do I perhaps have to change to Google Cloud Run instead to have the chance to run my own container there? See Python flask app using google cloud functions . It seems that large file sizes, which can certainly arise from SQL query results in csv, are not at all recommended in Cloud Functions, see Streaming binary data from Google Cloud Storage to Cloud Function , and its comment:

... I do not recommend using Cloud Functions to process large files though. I would rather recommend other serverless options as App Engine, or Cloud Run. Large files processing may take a while and Cloud Functions timeout after a certain time.

idea

And the next comment mentions GCSFS (Python module) which probably does not offer querying an external db though:

I did manage to get something working using GCSFS which allows me to open files in cloud storage and stream binary as if they were local.

Answer 1

If the database is on a VM, and in your VPC, you can create a VPC connector and attach it to your Cloud Function to access it.

If it's deployed else where,

• Either the database has a public IP, and Cloud Functions can directly access it.
• Or the database has a private IP and you need to create a VPN between your VPC and the private foreign network with your database. And again add a serverless VPC connector to Cloud Functions to allow it to your your VPC and the VPN to access the database.