How to determine whether PostgreSQL running in a Docker container is done initializing with Marten?

Question

I am basically writing a little bit of a program to benchmark the insertion performance of PostgreSQL over a given table growth, and I would like to make sure that when I am using Marten to insert data, the database is fully ready to accept insertions.

I am using Docker.DotNet to spawn a new container running the latest PostgreSQL image but even if the container is in a running state it is sometimes not the case for the Postgre running inside that container.

Of course, I could have added a Thread. Sleep Thread. Sleep but this is a bit random so I would rather like something deterministic to figure out when the database is ready to accept insertions?

public static class Program
{
    public static async Task Main(params string[] args)
    {
        const string imageName = "postgres:latest";
        const string containerName = "MyProgreSQL";

        var client = new DockerClientConfiguration(Docker.DefaultLocalApiUri).CreateClient();
        var containers = await client.Containers.SearchByNameAsync(containerName);

        var container = containers.SingleOrDefault();
        if (container != null)
        {
            await client.Containers.StopAndRemoveContainerAsync(container);
        }

        var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
        {
            Image = imageName,

            HostConfig = new HostConfig
            {
                PortBindings = new Dictionary<string, IList<PortBinding>>
                {
                    {"5432/tcp", new List<PortBinding>
                    {
                        new PortBinding
                        {
                            HostPort = "5432"
                        }
                    }}
                },
                PublishAllPorts = true
            },
            Env = new List<string>
            {
                "POSTGRES_PASSWORD=root",
                "POSTGRES_USER=root"
            },
            Name = containerName
        });

        var containerState = string.Empty;
        while (containerState != "running")
        {
            containers = await client.Containers.SearchByNameAsync(containerName);
            container = containers.Single();
            containerState = container.State;
        }

        var store = DocumentStore.For("host=localhost;database=postgres;password=root;username=root");

        var stopwatch = new Stopwatch();
        using (var session = store.LightweightSession())
        {
            var orders = OrderHelpers.FakeOrders(10000);
            session.StoreObjects(orders);
            stopwatch.Start();
            await session.SaveChangesAsync();
            var elapsed = stopwatch.Elapsed;
            // Whatever else needs to be done...
        }
    }
}

FYI, the if I am running the program above without pausing before the line await session.SaveChangesAsync(); I am running than into the following exception:

Unhandled Exception: Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.EndOfStreamException: Attempted to read past the end of the streams.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 154
   --- End of inner exception stack trace ---
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlReadBuffer.cs:line 175
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:l
ine 955
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.Auth.cs
:line 26
   at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\NpgsqlConne
ctor.cs:line 425
   at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\
npgsql\src\Npgsql\ConnectorPool.cs:line 246
   at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnection.Open() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 153
   at Marten.Storage.Tenant.generateOrUpdateFeature(Type featureType, IFeatureSchema feature)
   at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
   at Marten.Storage.Tenant.ensureStorageExists(IList`1 types, Type featureType)
   at Marten.Storage.Tenant.StorageFor(Type documentType)
   at Marten.DocumentSession.Store[T](T[] entities)
   at Baseline.GenericEnumerableExtensions.Each[T](IEnumerable`1 values, Action`1 eachAction)
   at Marten.DocumentSession.StoreObjects(IEnumerable`1 documents)
   at Benchmark.Program.Main(String[] args) in C:\Users\eperret\Desktop\Benchmark\Benchmark\Program.cs:line 117
   at Benchmark.Program.<Main>(String[] args)

[EDIT]

I accepted an answer but due to a bug about health parameters equivalence in the Docker.DotNet I could not leverage the solution given in the answer (I still think that a proper translation of that docker command in the .NET client, if actually possible) would be the best solution. In the meanwhile this is how I solved my problem, I basically poll the command that was expected to run in the health check aside until the result is ok:

Program.cs , the actual meat of the code:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Benchmark.DockerClient;
using Benchmark.Domain;
using Benchmark.Utils;
using Docker.DotNet;
using Docker.DotNet.Models;
using Marten;
using Microsoft.Extensions.Configuration;

namespace Benchmark
{
    public static class Program
    {
        public static async Task Main(params string[] args)
        {
            var configurationBuilder = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json");

            var configuration = new Configuration();
            configurationBuilder.Build().Bind(configuration);

            var client = new DockerClientConfiguration(DockerClient.Docker.DefaultLocalApiUri).CreateClient();
            var containers = await client.Containers.SearchByNameAsync(configuration.ContainerName);

            var container = containers.SingleOrDefault();
            if (container != null)
            {
                await client.Containers.StopAndRemoveContainerAsync(container.ID);
            }

            var createdContainer = await client.Containers.RunContainerAsync(new CreateContainerParameters
            {
                Image = configuration.ImageName,
                HostConfig = new HostConfig
                {
                    PortBindings = new Dictionary<string, IList<PortBinding>>
                    {
                        {$@"{configuration.ContainerPort}/{configuration.ContainerPortProtocol}", new List<PortBinding>
                        {
                            new PortBinding
                            {
                                HostPort = configuration.HostPort
                            }
                        }}
                    },
                    PublishAllPorts = true
                },
                Env = new List<string>
                {
                    $"POSTGRES_USER={configuration.Username}",
                    $"POSTGRES_PASSWORD={configuration.Password}"
                },
                Name = configuration.ContainerName
            });

            var isContainerReady = false;

            while (!isContainerReady)
            {
                var result = await client.Containers.RunCommandInContainerAsync(createdContainer.ID, "pg_isready -U postgres");
                if (result.stdout.TrimEnd('\n') == $"/var/run/postgresql:{configuration.ContainerPort} - accepting connections")
                {
                    isContainerReady = true;
                }
            }

            var store = DocumentStore.For($"host=localhost;" +
                                          $"database={configuration.DatabaseName};" +
                                          $"username={configuration.Username};" +
                                          $"password={configuration.Password}");

            // Whatever else needs to be done...
    }
}

Extensions being defined in ContainerOperationsExtensions.cs :

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Docker.DotNet;
using Docker.DotNet.Models;

namespace Benchmark.DockerClient
{
    public static class ContainerOperationsExtensions
    {
        public static async Task<IList<ContainerListResponse>> SearchByNameAsync(this IContainerOperations source, string name, bool all = true)
        {
            return await source.ListContainersAsync(new ContainersListParameters
            {
                All = all,
                Filters = new Dictionary<string, IDictionary<string, bool>>
                {
                    {"name", new Dictionary<string, bool>
                        {
                            {name, true}
                        }
                    }
                }
            });
        }

        public static async Task StopAndRemoveContainerAsync(this IContainerOperations source, string containerId)
        {
            await source.StopContainerAsync(containerId, new ContainerStopParameters());
            await source.RemoveContainerAsync(containerId, new ContainerRemoveParameters());
        }

        public static async Task<CreateContainerResponse> RunContainerAsync(this IContainerOperations source, CreateContainerParameters parameters)
        {
            var createdContainer = await source.CreateContainerAsync(parameters);
            await source.StartContainerAsync(createdContainer.ID, new ContainerStartParameters());
            return createdContainer;
        }

        public static async Task<(string stdout, string stderr)> RunCommandInContainerAsync(this IContainerOperations source, string containerId, string command)
        {
            var commandTokens = command.Split(" ", StringSplitOptions.RemoveEmptyEntries);

            var createdExec = await source.ExecCreateContainerAsync(containerId, new ContainerExecCreateParameters
            {
                AttachStderr = true,
                AttachStdout = true,
                Cmd = commandTokens
            });

            var multiplexedStream = await source.StartAndAttachContainerExecAsync(createdExec.ID, false);

            return await multiplexedStream.ReadOutputToEndAsync(CancellationToken.None);
        }
    }
}

Docker.cs to get the local docker api uri:

using System;
using System.Runtime.InteropServices;

namespace Benchmark.DockerClient
{
    public static class Docker
    {
        static Docker()
        {
            DefaultLocalApiUri = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) 
                ? new Uri("npipe://./pipe/docker_engine")
                : new Uri("unix:/var/run/docker.sock");
        }

        public static Uri DefaultLocalApiUri { get; }
    }
}

Answer 1

I suggest you to use a custom healtcheck to check if the database is ready to accept connections.

I am not familiar with the .NET client of Docker, but the following docker run command shows what you should try :

docker run --name postgres \
    --health-cmd='pg_isready -U postgres' \
    --health-interval='10s' \
    --health-timeout='5s' \
    --health-start-period='10s' \
    postgres:latest

Time parameters should be updated accordingly to your needs.

With this healtcheck configured, your application must wait for the container to be in state " healthy " before trying to connect to the database. The status " healthy ", in that particular case, means that the command pg_isready -U postgres have succeeded (so the database is ready to accept connections).

The status of your container can be retrieved with :

docker inspect --format "{{json .State.Health.Status }}" postgres