Cloud RUN job InterfaceError - (pg8000.exceptions.InterfaceError) - when connecting to cloud sql

Question

This question is about cloud run job (not services).

InterfaceError - (pg8000.exceptions.InterfaceError) Can't create a connection to host 127.0.0.1 and port 5432 (timeout is None and source_address is None).

I have python code that connects to cloud sql and runs a simple select * on sql db.

• My cloud sql instance is public, in same account & region as cloud run
• I had added cloud sql connection to cloud RUN job through console:

Recreating this error on local machine using docker:

When I run the container locally along with cloud sql proxy as shown below it works successfully:

docker run --rm --network=host job1

If I remove --network=host then I can recreate the exact error (shown in cloud RUN) locally:

docker run --rm job1

Am I using wrong host?

• On local machine I set host - 127.0.0.1 as shown in official example - gcp github
• On cloud RUN I tried setting host to 127.0.0.1 and /cloudsql/project:region:instance . Both did not work

My python code that runs on cloud RUN:

import os

import pandas
import sqlalchemy


def execute_sql(query, engine):
    with engine.connect() as connection:
        df = pandas.read_sql(
            con=connection,
            sql=query
        )
    return df


def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
    db_user = 'postgres'  # e.g. 'my-database-user'
    db_pass = 'abcxyz123'  # e.g. 'my-database-password'
    db_name = 'development'  # e.g. 'my-database'
    db_host = os.getenv('host', '127.0.0.1')
    db_port = os.getenv('port', 5432)
    connect_args = {}
    pool = sqlalchemy.create_engine(
        sqlalchemy.engine.url.URL.create(
            drivername="postgresql+pg8000",
            username=db_user,
            password=db_pass,
            host=db_host,
            port=db_port,
            database=db_name,
        ),
        connect_args=connect_args,
        pool_size=5,
        max_overflow=2,
        pool_timeout=30,  # 30 seconds
        pool_recycle=1800,  # 30 minutes
    )
    return pool


def func1():
    engine = connect_tcp_socket()
    query = 'select * from public.administrator;'
    df = execute_sql(query, engine)
    print(f'df={df}')


if __name__ == '__main__':
    func1()

Answer 1

How is your Cloud SQL instance configured? is it using private or public ip? Is the Cloud SQL instance in the same project, region and net? usually when you are connecting to 127.0.0.1 you are actually connecting to the Cloud SQL via Auth Proxy locally, however this doesn't apply for Cloud Run, depending on your cloud sql configuration you want to make sure that you configured the Cloud SQL connectivity at the deployement moment using the following flags if your Cloud SQL uses public ip

gcloud run deploy
--image=IMAGE
--add-cloudsql-instances=INSTANCE_CONNECTION_NAME

If your Cloud SQL is using private ip you want to use the instance private ip and not 127.0.0.1

Answer 2

I was unable to connect to proxy from container running on cloud run jobs. So instead I started proxy manually from inside the Dockerfile. This way I know exact port and host to map.

To run the python script job1.py shown in the question use following files

Dockerfile:

FROM python:buster
ADD . /code

RUN pip install --upgrade pip
RUN pip install pandas sqlalchemy
RUN pip install pg8000 cloud-sql-python-connector

# download the cloudsql proxy binary
RUN mkdir "/workspace"
RUN wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /workspace/cloud_sql_proxy
RUN chmod +x /workspace/cloud_sql_proxy

RUN chmod +x /code/start.sh
CMD ["/code/start.sh"]

start.sh

#!/bin/sh
/workspace/cloud_sql_proxy -instances=pangea-dev-314501:us-central1:pangea-dev=tcp:5432 -credential_file=/key/sv_account_key.json &
sleep 6
python /code/job1.py  

Answer 3

I'd recommend using the Cloud SQL Python Connector package as it offers a consistent way of connecting across all environments (Local machine, Cloud Run, App Engine, Cloud Functions etc.) and provides the following benefits (one of which is not having to worry about IP addresses or needing the Cloud SQL proxy):

• IAM Authorization: uses IAM permissions to control who/what can connect to your Cloud SQL instances
• Improved Security: uses robust, updated TLS 1.3 encryption and identity verification between the client connector and the server-side proxy, independent of the database protocol.
• Convenience: removes the requirement to use and distribute SSL certificates, as well as manage firewalls or source/destination IP addresses.
• (optionally) IAM DB Authentication: provides support for Cloud SQL's automatic IAM DB AuthN feature.

You can find a Flask App example using the Python Connector in the same Github repo you linked in your question.

Basic usage example:

from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy

# build connection (for creator argument of connection pool)
def getconn():
    # Cloud SQL Python Connector object
    with Connector() as connector:
        conn = connector.connect(
            "project:region:instance",  # Cloud SQL instance connection name
            "pg8000",
            user="my-user",
            password="my-password",
            db="my-db-name",
            ip_type=IPTypes.PUBLIC  # IPTypes.PRIVATE for private IP
        )
    return conn

# create connection pool
pool = sqlalchemy.create_engine(
    "postgresql+pg8000://",
    creator=getconn,
)