Container exposed port can only be accessed via *external* IP address, not localhost

Question

There are so many posted questions in this area, but my case is the opposite: I have a docker container with an exposed port that does work using the external IP address, but doesn't work via localhost or 127.0.0.1 . I can't figure out why. I think the cause may be the same that's causing my real issue, which is that my docker-compose network of containers cannot communicate with each other. This is on Ubuntu 22.10.

Here is the simple docker-compose.yml file:

version: '3.1'
services:
  zcashd:
    image: electriccoinco/zcashd
    restart: unless-stopped
    ports:
    - 8232:8232 # RPC (security sensitive)
    - 8233:8233 # Zcash network (public)
    volumes:
    - data:/srv/zcashd/.zcash
    - params:/srv/zcashd/.zcash-params

With that container running, things look good:

docker ps
CONTAINER ID   IMAGE                   COMMAND            CREATED       STATUS             PORTS                                                           NAMES
73cbbeb622e2   electriccoinco/zcashd   "/entrypoint.sh"   9 hours ago   Up About an hour   0.0.0.0:8232-8233->8232-8233/tcp, :::8232-8233->8232-8233/tcp   zcash_zcashd_1

But outside the container, I can only access port 8232 via the docker host's own IP address. Note how using localhost fails:

curl --user zecwallet -d '{"jsonrpc":"1.0","id":1,"method":"getblockchaininfo","params":[]}' -H 'Content-Type:text/plain' http://localhost:8232 -v
Enter host password for user 'zecwallet':
*   Trying 127.0.0.1:8232...
* Connected to localhost (127.0.0.1) port 8232 (#0)
* Server auth using Basic with user 'zecwallet'
> POST / HTTP/1.1
> Host: localhost:8232
> Authorization: Basic *redacted*
> User-Agent: curl/7.85.0
> Accept: */*
> Content-Type:text/plain
> Content-Length: 65
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Date: Tue, 13 Dec 2022 14:42:48 GMT
< Content-Length: 0
< Content-Type: text/html; charset=ISO-8859-1
< 
* Connection #0 to host localhost left intact

And note how it succeeds in the same shell but using the host's IP address:

curl --user zecwallet -d '{"jsonrpc":"1.0","id":1,"method":"getblockchaininfo","params":[]}' -H 'Content-Type:text/plain' http://192.168.0.118:8232 -v
Enter host password for user 'zecwallet':
*   Trying 192.168.0.118:8232...
* Connected to 192.168.0.118 (192.168.0.118) port 8232 (#0)
* Server auth using Basic with user 'zecwallet'
> POST / HTTP/1.1
> Host: 192.168.0.118:8232
> Authorization: Basic *redacted*
> User-Agent: curl/7.85.0
> Accept: */*
> Content-Type:text/plain
> Content-Length: 65
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Tue, 13 Dec 2022 14:45:56 GMT
< Content-Length: 2106
< 
{"result":{...},"error":null,"id":1}

It also works when the HTTP request comes from a different machine, which of course uses the host's IP address as well.

So, that's weird and a little irritating that localhost doesn't work. But what is really blocking me is when I add another service to my docker-compose.yml file. It tries to access my zcashd service via that host name and gets the same 403 Forbidden response that I'm getting from outside the container when using localhost. These containers are supposed to be able to talk to each other, so why isn't it working?

I have not set up ufw or any other firewall on this machine at this point. Here's a bit more info:

# docker inspect zcash_zcashd_1
...
        "NetworkSettings": {
            "Bridge": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8232/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8232"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8232"
                    }
                ],
                "8233/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8233"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8233"
                    }
                ]
            },
...
           "Networks": {
                "zcash_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "73cbbeb622e2",
                        "zcashd"
                    ],
                    "NetworkID": "...",
                    "EndpointID": "...",
                    "Gateway": "172.25.0.1",
                    "IPAddress": "172.25.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "...",
                    "DriverOpts": null
                }
            }

Answer 1

It turns out this is is not a docker problem. It's a problem with my zcash.conf configuration file within the container itself. It has an AllowedIPs list which rejected incoming requests based on the IP address. It is supposed to be filtering based on the IP address of the source, but evidently the IP address that the source uses to talk to it influences how the target sees the source IP as well.

Adding this to my zcash.conf file resolved the issue:

rpcallowip=172.0.0.0/255.0.0.0
rpcallowip=127.0.0.1/255.255.255.255

Docker always assigns IP addresses that start with 172, so the first line allowed access from other containers, and the second line made it available via localhost on the host as well.