Read bytea data is slow in PostgreSQL

Question

I store data in bytea column in PostgreSQL 9.5 database on Windows.

The data transmission speed is lower than I expect : about 1.5mb per second.

The following code

        using (var conn = ConnectionProvider.GetOpened())
        using (var comm = new NpgsqlCommand("SELECT mycolumn FROM mytable", conn))
        using (var dr = comm.ExecuteReader())
        {
            var clock = Stopwatch.StartNew();
            while (dr.Read())
            {
                var bytes = (byte[])dr[0];
                Debug.WriteLine($"bytes={bytes.Length}, time={clock.Elapsed}");
                clock.Restart();
            }
        }

Produces the following output

bytes=3895534, time=00:00:02.4397086
bytes=4085257, time=00:00:02.7220734
bytes=4333460, time=00:00:02.4462513
bytes=4656500, time=00:00:02.7401579
bytes=5191876, time=00:00:02.7959250
bytes=5159785, time=00:00:02.7693224
bytes=5184718, time=00:00:03.0613514
bytes=720401, time=00:00:00.0227767
bytes=5182772, time=00:00:02.7704914
bytes=538456, time=00:00:00.2996142
bytes=246085, time=00:00:00.0003131
Total: 00:00:22.5199268

The strange thing is that reading last 246kb took less than a millisecond, and reading 720kb in the middle took just 22ms.

Is the reading speed 5mb per 3 sec normal? How I can increase the reading speed?

Details.

My application starts PostgreSQL server on startup and shut downs it on exit.

I start server with the following code

public static void StartServer(string dataDirectory, int port)
{      
    Invoke("pg_ctl", $"start -w  -D \"{dataDirectory}\" -m fast -o \"-B 512MB -p {port} -c temp_buffers=32MB -c work_mem=32MB\"");
}

Also, i change the storage type to my column :

ALTER TABLE mytable ALTER COLUMN mycolumn SET STORAGE EXTERNAL;

I use npgsql 3.0.4.0 and PostgreSQL 9.5 on Windows 10

Answer 1

Running here with Npgsql 3.0.8 (should be the same), PostgreSQL 9.5, windows 10 and Npgsql I don't get your results at all:

bytes=3895534, time=00:00:00.0022591
bytes=4085257, time=00:00:00.0208912
bytes=4333460, time=00:00:00.0228702
bytes=4656500, time=00:00:00.0237144
bytes=5191876, time=00:00:00.0317834
bytes=5159785, time=00:00:00.0268229
bytes=5184718, time=00:00:00.0159028
bytes=720401, time=00:00:00.0130150
bytes=5182772, time=00:00:00.0153306
bytes=538456, time=00:00:00.0021693
bytes=246085, time=00:00:00.0005174

First, what server are you running against, is it on localhost or on some remote machine?

The second thing that comes to mind, is you stopping and starting the server as part of the test. The slow performance you're seeing may be part of a warm-up on PostgreSQL's side. Try to remove it and see if the results can be reproduced after several times of running your tests.

Otherwise this looks like some environmental problem (client machine, server machine or network). I'd try to reproduce the problem on a different machine or in a different setting and go from there.