archive_cleanup_command 不清除存档的 wal 文件

Question

Main question:
archive_cleanup_command in the postgresql.conf file does not clear the archived wal files. How can I get it to clear the archived wal files?

Relevant information:

• My OS is Linux, Ubuntu v18.04 LTS.
• Database is Postgresql version 13

My current settings:
/etc/postgresql/13/main/postgresql.conf file:

wal_level = replica
wal_compression = on

wal_recycle = on
checkpoint_timeout = 5min

max_wal_size = 1GB
min_wal_size = 80MB

archive_mode = on
archive_command = 'pxz --compress --keep --force -6 --to-stdout --quiet %p > /datadrive/postgresql/13/wal_aerchives/%f.xz'

archive_timeout = 10min

restore_command = 'pxz --decompress --keep --force -6 --to-std-out --quiet /datadrive/postgresql/13/wal_archives/%f.xz > %p'

archive_cleanup_command = 'pg_archivecleanup -d -x .xz /datadrive/postgresql/13/wal_archives %r >> /datadrive/postgresql/13/wal_archives/archive_cleanup_command.log 2>&1'

archive_cleanup_command.log has 777 permissions.

I have a master database doing logical replication with a publication and a slave database subscribing to that publication. It is on the slave that I am intending to do the archiving and restore points.

What am I expecting to happen?
The checkpoint timeout setting in the postgresql.conf file means that a restart point is created atleast every 5 mins. And the archive_timeout setting of 10 mins means that postgresql forces a logfile segment switch after every 10 mins. Therefore, atleast every 10 mins, a restart point is created. Whenever a restart point is created, the archive cleanup command is run. When this command is run it will clear all the .xz files older than this restart point. Therefore the wal_archives directory should not really have .xz files older than 20mins or even 2hours....

What is actually happening?

• The /datadrive/postgresql/13/wal_archives directory piles up with lots of .xz files that never get cleared.

• cat archive_cleanup_command.log shows an empty file. Nothing is ever writing to it.

• When I run the pg_archivecleanup command manually via bash, it works (ie clears all the archive files before the one specified and cat archive_cleanup_command shows the files that were cleared.
  Example:

   pg_archivecleanup -d -x .xz /datadrive/postgresql/13/wal_archives 000000010000045E000000E5 >> /datadrive/postgresql/13/wal_archives/archive_cleanup_command.log 2>&1

  Then running cat archive_cleanup_command.log gives this:

   pg_archivecleanup: keeping WAL file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E5" and later pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000DE.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000DF.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E0.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E1.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E2.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E3.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E4.xz"

What have I tried?

• I have tried various permission settings (examples: chmod 777 the wal_archive directory, add other users to the postgres group, etc...)

• Extensively and thoroughly read the postgresql documentation and looked atleast 20 different related stackoverflow posts.

• Initially tried 7zip cmd line tool to do the zipping instead of pxz.

• Successfully restarted the database multiple times using the following commands:

   sudo systemctl stop postgresql@13-main sudo systemctl start postgresql@13-main

• Dropped the logical replication and re-created the publication on the master and subscription on the slave.

• Enabled checkpoints on the master itself.

• Looked at /var/log/postgresql/postgresql-13-main.log . Unfortunately no relevant errors show up in this log.

Answer 1

Restartpoints, restore_command and archive_cleanup_command only apply to streaming ("physical") replication, or to recovery in general, not to logical replication.

A logical replication standby is not in recovery, it is open for reading and writing. In that status, recovery settings like archive_cleanup_command are ignored.

You will have to find another mechanism to delete old WAL archives, ideally in combination with your backup solution.