Handling backups of a large table (>1 TB) in Postgres?

Question

I have a 1TB table (X) that is a pain to backup.

The table X contains historical log data that is not often updated after creation. We usually only access a single row at a time, so performance is still very good.

We currently make nightly full logical backups, and exclude X for the sake of backup time and space. We do not need historical backups of X, since the log files from which it is populated are backed up themselves. However, recovery of X by re-processing of the log files would take an unnecessary long time.

I'd like to include X in our backup strategy so that our recovery time can be much faster. It doesn't seem feasible to include X in the nightly logical backup.

Ideally, I'd like a single full backup for X that is updated incrementally (purely to save time).

I lack the experience to investigate solutions alone, and I'm wondering what my options are?

Barman for incremental updates? Partition X? Both?

After doing some more reading, I'm inclined to partition the table and write a nightly script to perform logical backups only on the changed table partitions (and replace the previous backups). However, this strategy may still take a long time during recovery with a pg_restore ... Thoughts?

Thanks!

Answer 1

I think using barman with the rsync/SSH + WAL streaming option and performing incremental backups is the best option in your case. Going this way makes your nightly backups easier & less costly, since you don't have to do much logic yourself once you configure barman. I will update this with my blog shortly that details the steps.

Logical backups may not be the right approach for periodic backups when dealing with large databases. When using physical backups even though your backup size is large its more than compensated in the acquisition & restore cost (performance, speed & simplicity).

Thanks

UPDATE (2020-08-27):

Below is a git repo with the end-end demonstration. There are many versions of implementations out there that have done it, but if you wanted to do something from the scratch & keep the image simple (avoiding unnecessary dependencies), please take a look at this implementation,

https://github.com/softwarebrahma/PostgreSQL-Disaster-Recovery-With-Barman

Thanks