Flask Docker container not connecting to another Postgres docker container

Question

So I built an app on my home machine and it worked perfectly. Unfortunately moving the configuration of three docker containers where one flask container accesses another postgres server fails. I am accessing the postgres container using psycopg2 using a hostname. The issue has been isolated to the connecting between one docker container to the next as I can access the server itself, the backend works if it doesn't have to access to the server, but it fails when it does. So heres the config I have currently (I am happy to send any config needed but in the interest of not sending an infinite amount Ill start with just docker compose):

Here is my docker compose that works on a intel macbook pro with docker compose v2.0.0 but doesnt work on an ampere oracle vm with docker compose v2.1.1

services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile.api
    # image: react-flask-app-api
    depends_on:
      - postgres_real
    ports:
      - "5000:5000"
    links:
      - postgres_real:postgres_real
    networks:
      - backend
  client:
    build:
      context: .
      dockerfile: Dockerfile.client
    depends_on:
      - backend
    ports:
      - "3000:80"
    links:
      - backend:backend
    networks:
      - backend
  postgres_real:
    container_name: postgres_real
    image: postgres:latest
    restart: always
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
      PGDATA: /var/lib/postgresql/data/pgdata
    ports:
      - "5432:5432"
    expose:
      - "5432"
    networks:
      - backend
networks:
  backend:
    driver: bridge

EDIT: So I would like to simplify my question further. After more testing, curl commands and attempting to forward requests through nginx using the following configuration

# nginx configuration for Docker

server {
    listen       80;
    server_name  _;

    root   /usr/share/nginx/html;
    index index.html;
    error_page   500 502 503 504  /50x.html;

    location / {
        try_files $uri $uri/ =404;
        add_header Cache-Control "no-cache";
    }

    location /static {
        expires 1y;
        add_header Cache-Control "public";
    }

    location /api {
        proxy_pass http://backend:5000;
    }
}

I have found that on a oracle server, the docker su.net does not actually work in that it is not able to make a request from one docker container to another. I am not sure what is needed so here are the probable required configs:

Config of my docker.network that works on my personal machine but not on the server

   {
        "Name": "congress_backend",
        "Id": "b484525801f859d3f38b59ea3b74baa6214e12acf3fcdf72e2115ee1a7c7c192",
        "Created": "2022-12-24T07:50:54.96602112Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.23.0.0/16",
                    "Gateway": "172.23.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "188de200ef405c8e1b859c57fcecda64c7b951ca38cd17d92db301b2b63677a7": {
                "Name": "congress-client-1",
                "EndpointID": "a170d1a21231a17c3c367e10dc0daaf56281bb9051161cae67e6e696c9f44ade",
                "MacAddress": "02:42:ac:17:00:04",
                "IPv4Address": "172.23.0.4/16",
                "IPv6Address": ""
            },
            "5fae87555e453a2d6e6d2d080d1c128932a90dc2e71c75b31a82b1a365713497": {
                "Name": "congress-backend-1",
                "EndpointID": "63801907b9c92b6ac318bbe6cf69e6b08a5f0b82e9fcb1590977116fce8f4690",
                "MacAddress": "02:42:ac:17:00:03",
                "IPv4Address": "172.23.0.3/16",
                "IPv6Address": ""
            },
            "73f9ece1a79e94e7442d01823f181e244a952f2bf3fde2e205e4c95a8aa74429": {
                "Name": "postgres_real",
                "EndpointID": "3d074029759e069f66519a0f3308d50801d1e1c09a48b1fa88870533980f6e45",
                "MacAddress": "02:42:ac:17:00:02",
                "IPv4Address": "172.23.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "backend",
            "com.docker.compose.project": "congress",
            "com.docker.compose.version": "2.1.1"
        }
    }

And here is my Iptables config for IPV4 rulesv4:

# CLOUD_IMG: This file was created/modified by the Cloud Image build process
# iptables configuration for Oracle Cloud Infrastructure

# See the Oracle-Provided Images section in the Oracle Cloud Infrastructure
# documentation for security impact of modifying or removing these rule

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [463:49013]
:InstanceServices - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 51820 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5000 -j ACCEPT
-A INPUT -i docker0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.4.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.5.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp --dport 123 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with icmp-port-unreachable
COMMIT

My application is configured using a flask application with something like this

@app.route('/api/bill/<bill_slug>')
def bill_data(bill_slug):
    data = get_bill_data(bill_slug.upper())
    return jsonify(data)
# get bill data uses psycopg2 like this:
conn = psycopg2.connect(
            host=os.environ.get('DB_HOST'),
            database=os.environ.get('POSTGRES_DB'),
            user=os.environ.get('POSTGRES_USER'),
            password=os.environ.get('POSTGRES_PASSWORD')
        )
# where DB_HOST is in this format postgresql://user:password@postgres_real:5432/database_name

if I try a curl command on something that just returns the time it works but if I try something similar like

curl 0.0.0.0:5000/api/bill/Hello

it returns with

curl: (52) Empty reply from server

internally in docker it errors as

[2022-12-24 16:19:37 +0000] [8] [CRITICAL] WORKER TIMEOUT (pid:11)
[2022-12-24 16:19:38 +0000] [8] [WARNING] Worker with pid 11 was terminated due to signal 9
[2022-12-24 16:19:38 +0000] [12] [INFO] Booting worker with pid: 12

Answer 1

If anyone is wondering how this is solved, I recommend checking out your iptables configuration and seeing if there is a legacy configuration as well. In my case, the legacy iptables configuration blocked all forwarding. After turning this off, docker worked perfectly.