Cannot connect to a Postgres SQL instance from inside a pod within a GKE

Question

I am new to Terraform and Google Cloud.

I am trying to create a K8 cluster where pods inside the cluster could communicate with a Postgres server I have within the same VPC.

However, when I try to connect to the server using a psql client within a ubuntu pod inside the cluster using psql -h <PRIVATE_IP_OF_POSTGRES_SERVER> -U postgresadmin , it waits for a long time and throws the following timeout error.

psql: could not connect to server: Connection timed out
        Is the server running on host "<PRIVATE_IP_OF_POSTGRES_SERVER>" and accepting
        TCP/IP connections on port 5432?

I even tried to telnet <PRIVATE_IP_OF_POSTGRES_SERVER> and that is also not working. (PING is not working as well)

Here is my terraform script for creating the VPC and the K8 cluster:

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

variable "gke_username" {
  default     = ""
  description = "gke username"
}

variable "gke_password" {
  default     = ""
  description = "gke password"
}

variable "gke_num_nodes" {
  default     = 1
  description = "number of gke nodes"
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# VPC
resource "google_compute_network" "vpc" {
  name                    = "${var.project_id}-vpc"
  auto_create_subnetworks = "false"
}

# Subnet
resource "google_compute_subnetwork" "subnet" {
  name          = "${var.project_id}-subnet"
  region        = var.region
  network       = google_compute_network.vpc.name
  ip_cidr_range = "10.10.0.0/24"
}

# GKE cluster
resource "google_container_cluster" "primary" {
  name     = "${var.project_id}-gke"
  location = var.region

  remove_default_node_pool = true
  initial_node_count       = 1

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name

  # Create a VPC-native cluster by configuring `ip_allocation_policy` 
  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "/16"
    services_ipv4_cidr_block = "/22"
  }

  master_auth {
    username = var.gke_username
    password = var.gke_password

    client_certificate_config {
      issue_client_certificate = false
    }
  }
}

# Separately Managed Node Pool
resource "google_container_node_pool" "primary_nodes" {
  name       = "${google_container_cluster.primary.name}-node-pool"
  location   = var.region
  cluster    = google_container_cluster.primary.name
  node_count = var.gke_num_nodes

  node_config {
    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]

    labels = {
      env = var.project_id
    }

    # preemptible  = true
    machine_type = "n1-standard-1"
    tags         = ["gke-node", "${var.project_id}-gke"]
    metadata = {
      disable-legacy-endpoints = "true"
    }
  }
}

Here is my terraform script file for creating the Postgres server:

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "3.61.0"
    }
  }
}

data "google_compute_network" "my_network" {
  name = "novade-lite-vpc"
}

provider "google" {
  project = var.project_id
  region  = var.region
}

resource "random_id" "db_name_suffix" {
  byte_length = 4
}

resource "google_compute_global_address" "private_ip_address" {
  name          = "private-ip-address"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 16
  network       = data.google_compute_network.my_network.self_link
}

resource "google_service_networking_connection" "private_vpc_connection" {
  network                 = data.google_compute_network.my_network.self_link
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}

resource "google_sql_database_instance" "postgres" {
  name             = "postgres-instance-${random_id.db_name_suffix.hex}"
  database_version = "POSTGRES_11"
  depends_on = [google_service_networking_connection.private_vpc_connection]

  settings {
    tier = "db-f1-micro"

    ip_configuration {
      ipv4_enabled    = false
      private_network = data.google_compute_network.my_network.self_link
    }
  }
}

resource "google_sql_database" "default" {
  name       = "default"
  project    = var.project_id
  instance   = google_sql_database_instance.postgres.name
  collation  = "en_US.UTF8"
  depends_on = [google_sql_database_instance.postgres]
}

resource "google_sql_user" "users" {
  name     = "postgresadmin"
  instance = google_sql_database_instance.postgres.name
  password = "password"
  depends_on = [google_sql_database_instance.postgres]
}

Here is my variables file (Same for both the above scripts):

project_id = "test-project"
region     = "europe-west2"

Any help is much appreciated!

Answer 1

I am not aware of how GKE setup works but I can give u some pointers based on my understanding of similar setup in AWS.

Your cluster pod will be one on of the NODES, this node will be part of some subnet and also will have some security group (type of firewall) attached to it.

Similarly postgres DB is created in some subnet and security group attached to it.

Now make sure that the DB security group all inbound traffic on 5432 port from pods subnet range or from security group.