Postgres:10 in docker swarm cluster. Database system is shut down

Question

I use postgres:10 ( https://hub.docker.com/_/postgres/ ) image for DB. It is deployed in docker swarm cluster.

After running DB replica I got database system is shut down in DB's log.

2018-05-11 10:26:53.073 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 10:26:53.073 UTC [1] LOG:  listening on IPv6 address "::", port 5432,
2018-05-11 10:26:53.077 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 10:26:53.092 UTC [20] LOG:  database system was shut down at 2018-05-11 10:26:20 UTC,
2018-05-11 10:26:53.100 UTC [1] LOG:  database system is ready to accept connections,
The files belonging to this database system will be owned by user "postgres".,
This user must also own the server process.,
,
The database cluster will be initialized with locale "en_US.utf8".,
The default database encoding has accordingly been set to "UTF8".,
The default text search configuration will be set to "english".,
,
Data page checksums are disabled.,
,
fixing permissions on existing directory /var/lib/postgresql/data ... ok,
creating subdirectories ... ok,
selecting default max_connections ... 100,
selecting default shared_buffers ... 128MB,
selecting dynamic shared memory implementation ... posix,
creating configuration files ... ok,
running bootstrap script ... ok,
performing post-bootstrap initialization ... ok,
,
WARNING: enabling "trust" authentication for local connections,
You can change this by editing pg_hba.conf or using the option -A, or,
--auth-local and --auth-host, the next time you run initdb.,
syncing data to disk ... ok,
,
Success. You can now start the database server using:,
,
    pg_ctl -D /var/lib/postgresql/data -l logfile start,
,
waiting for server to start....2018-05-11 09:39:21.129 UTC [37] LOG:  listening on IPv4 address "127.0.0.1", port 5432,
2018-05-11 09:39:21.130 UTC [37] LOG:  could not bind IPv6 address "::1": Cannot assign requested address,
2018-05-11 09:39:21.130 UTC [37] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.,
2018-05-11 09:39:21.133 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.147 UTC [38] LOG:  database system was shut down at 2018-05-11 09:39:20 UTC,
2018-05-11 09:39:21.152 UTC [37] LOG:  database system is ready to accept connections,
 done,
server started,
CREATE DATABASE,
,
CREATE ROLE,
,
,
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*,
,
2018-05-11 09:39:21.595 UTC [37] LOG:  received fast shutdown request,
waiting for server to shut down....2018-05-11 09:39:21.596 UTC [37] LOG:  aborting any active transactions,
2018-05-11 09:39:21.598 UTC [37] LOG:  worker process: logical replication launcher (PID 44) exited with exit code 1,
2018-05-11 09:39:21.599 UTC [39] LOG:  shutting down,
2018-05-11 09:39:21.613 UTC [37] LOG:  database system is shut down,
 done,
server stopped,
,
PostgreSQL init process complete; ready for start up.,
,
2018-05-11 09:39:21.706 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432,
2018-05-11 09:39:21.706 UTC [1] LOG:  listening on IPv6 address "::", port 5432,
2018-05-11 09:39:21.709 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2018-05-11 09:39:21.724 UTC [64] LOG:  database system was shut down at 2018-05-11 09:39:21 UTC,
2018-05-11 09:39:21.729 UTC [1] LOG:  database system is ready to accept connections,
2018-05-11 10:26:20.444 UTC [1] LOG:  received smart shutdown request,
2018-05-11 10:26:20.449 UTC [1] LOG:  worker process: logical replication launcher (PID 70) exited with exit code 1,
2018-05-11 10:26:20.449 UTC [65] LOG:  shutting down,
2018-05-11 10:26:20.460 UTC [1] LOG:  database system is shut down,

Image :

  FROM postgres:10

  COPY healthcheck /usr/local/bin/

  RUN chmod +x /usr/local/bin/healthcheck

  HEALTHCHECK --interval=30s --timeout=30s --retries=3 \
   CMD healthcheck

Snippet from docker-compose :

  db_jackrabbit:
    build: ./images/pgsql_jackrabbit
    container_name: db_jackrabbit
    environment:
      - POSTGRES_DB=${JACK_POSTGRES_DB}
      - POSTGRES_USER=${JACK_POSTGRES_USER}
      - POSTGRES_PASSWORD=${JACK_POSTGRES_PASSWORD}
    volumes:
      -  pgsql_jackrabbit_local:/var/lib/postgresql/data
    ports:
      - ${PORT_DB_JACKRABBIT}:5432

healthcheck:

#!/bin/bash
set -eo pipefail

host="$(hostname -i || echo '127.0.0.1')"
user="${POSTGRES_USER:-postgres}"
db="${POSTGRES_DB:-$POSTGRES_USER}"
export PGPASSWORD="${POSTGRES_PASSWORD:-}"

args=(
   # force postgres to not use the local unix socket (test "external" connectibility)
   --host "$host"
   --username "$user"
   --dbname "$db"
   --quiet --no-align --tuples-only
)

if select="$(echo 'SELECT 1' | psql "${args[@]}")" && [ "$select" = '1' ]; then
   exit 0
fi

exit 1

But DB still alive. It is shutdown periodically and accept connections again ( What it the problem? Thanks in advance!

Answer 1

Ok, so I solved my issue. This issue helped me .

It seems like Postgres initialization process does stop the initialization process once done, and it is another process which follows up and accepts connexion.

Hence I had:

  postgres:
    deploy:
      restart_policy:
        condition: on-failure
        window: 15m

and apparently docker received a end-of-process status code, so it stopped without going to the next process so never accepting connexions.

My interpretation may be incorrect, but at least if you face the issue, try removing restart_policy key to see if it fixes it.

I haven't tried to restore healthcheck yet, as it may also have undesired side effects.