WAL Archive hangs in postgres when gzip is used

Question

I have enabled the WAL Archiving with following archive command:

wal_keep_segments = 32
archive_mode = on
archive_command = 'gzip < %p > /mnt/nfs/archive/%f'

and on Slave I have restore command as:

restore_command = 'gunzip < /mnt/nfs/archive/%f > %p'
archive_cleanup_command = '/opt/PostgreSQL/9.4/bin/pg_archivecleanup -d /mnt/nfs/archive %r'

on Master I could see that many files are stuck. around 327 files are yet to be archived. Ideally it should be only 32 only.

the px command shows:

-bash-4.1$ ps x
  PID TTY      STAT   TIME COMMAND
 3302 ?        S      0:00 /opt/PostgreSQL/9.4/bin/postgres -D /opt/PostgreSQL/9.4/data
 3304 ?        Ss     0:00 postgres: logger process
 3306 ?        Ss     0:09 postgres: checkpointer process
 3307 ?        Ss     0:00 postgres: writer process
 3308 ?        Ss     0:06 postgres: wal writer process
 3309 ?        Ss     0:00 postgres: autovacuum launcher process
 3311 ?        Ss     0:00 postgres: stats collector process
 3582 ?        S      0:00 sshd: postgres@pts/1
 3583 pts/1    Ss     0:00 -bash
 3628 ?        Ss     0:00 postgres: archiver process   archiving 000000010000002D000000CB
 3673 ?        S      0:00 sh -c gzip < pg_xlog/000000010000002D000000CB > /mnt/nfs/archive/000000010000002D000000CB
 3674 ?        D      0:00 gzip
 3682 ?        S      0:00 sshd: postgres@pts/0
 3683 pts/0    Ss     0:00 -bash
 4070 ?        Ss     0:00 postgres: postgres postgres ::1(34561) idle
 4074 ?        Ss     0:00 postgres: postgres sorriso ::1(34562) idle
 4172 pts/0    S+     0:00 vi postgresql.conf
 4192 pts/1    R+     0:00 ps x
-bash-4.1$ ls | wc -l
327
-bash-4.1$

Answer 1

gzip and gunzip without flags expect to work with files, compressing or uncompressing them in-place. You're trying to use them as stream processors. That's not going to work.

You want to use gzip -c and zcat (or gunzip -c ) to tell them to use stdio.

Additionally, though, you should probably use a simple script as the archive_command that:

• Writes with gzip -c to a temp file
• Moves the temp file to the final location with mv

This ensures that the file is not read by the replica until it's fully written by the master.

Also, unless the master and replica are sharing the same network file system (or are both on the same host), you might actually need to use scp or similar to transfer the archive files. The restore_command uses paths on the replica , not on the master, so unless the replica server can access the WAL archive via NFS/CIFS/etc, you're going to need to copy the files.