Connecting to Google Cloud SQL instance on private IP from a VM with both private and public IPs fails

Question

What I'm trying to set up:

• Cloud SQL instance with private IP, Postgresql database
• A VM with one public IP and one private IP on same VPC network as the SQL instance is on (VM, SQL instance and VPC are all in the same region)
• VM has a service account with sufficient Cloud SQL client/viewer permissions
• SQL proxy on VM connecting to SQL instance. I run it with the -ip_address_types=PRIVATE argument I've found in some of the documentation.

Configuration code

Slightly simplified Terraform code for reproducing the state that confuses me is here: https://github.com/hallvors/gcp-network-issue-demo To test this, do the following:

1. create a new throwaway Google Cloud project.
2. For your convenience, you can run bootstrap.sh to enable the right services (it will ask for the ID of the Google project and assume you have a gcloud client which is logged in and has access).
3. Create a service account in the project, just make it owner for convenience, and save a key file in ./local-secrets/google-project-credentials.json
4. Update terraform.tfvars with project ID and e-mail of service account
5. terraform workspace new staging
6. terraform init
7. terraform apply

When Terraform is done, you should have a database and a VM set up in the project.

1. SSH into the VM and run sudo apt install postgresql-client-common postgresql-client
2. Look up the IP address of the DB instance
3. Run this (modify details as needed) psql --host 10.167.0.3 -U gcp-network-issue-demo-staging-db-user gcp-network-issue-demo-staging-database

What happens?

• Any attempt to actually use the connection, from for example psql client or db-migrate, times out
• If I remove the VM's public IP address from the setup, it connects fine. However, I need a publicly accessible VM for other services to connect to it..

What am I missing?

Answer 1

The cause of this problem was me failing to understand that a network interface can have both public and private IPs/networks. So my code set up one interface for the public and one for the private network for the google_compute_instance:

  # Update VM needs a public IP
  network_interface {
    network = "default"
    access_config {
    }
  }

  network_interface {
    network    = var.network
    subnetwork = var.subnetwork
  }

Now, I still don't fully understand networking but it appears you can not (easily?) specify what interface the database connection attempts should use and it does not automatically pick the right one. The fix is in this commit, configuring both access to the private network and the public one in one network interface:

https://github.com/hallvors/gcp-network-issue-demo/commit/ea14174c1087c89b92310b5b4913e12a4e17130d