Improve PostgreSQL pg_restore Performance from 130 hours

Question

I am trying to improve the time taken to restore a PostgreSQL database backup using pg_restore . The 29 GB gzip-compressed backup file is created from a 380 GB PostgreSQL database using pg_dump -Z0 -Fc piped into pigz .

During pg_restore , the database size is increasing at a rate of 50 MB/minute estimated using the SELECT pg_size_pretty(pg_database_size()) query. At this rate, it will take approximately 130 hours to complete the restore which is a very long time.

On further investigation, it appears that the CPU usage is low despite setting pg_restore to use 4 workers.

The disk write speed and IOPS are also very low:

Benchmarking the system's IO using fio has shown that it can do 300 MB/s writes and 2000 IOPS, so we are utilizing only about 20% of the potential IO capabilities.

Is there any way to speed up the database restore?

System

• Ubuntu 18.04.3
• 1 vCPU, 2 GB RAM, 4GB Swap
• 500 GB ZFS (2-way mirror array)
• PostgreSQL 11.6
• TimescaleDB 1.60

Steps taken to perform restore:

1. Decompress the .gz file to /var/lib/postgresql/backups/backup_2020-02-29 (~ 40mins)

2. Modify postgresql.conf settings

work_mem = 32MB
shared_buffers = 1GB            
maintenance_work_mem = 1GB      
full_page_writes = off
autovacuum = off
wal_buffers = -1

3. pg_ctl restart

4. Run the following commands inside psql :

CREATE DATABASE database_development;
\c database_development
CREATE EXTENSION timescaledb;
SELECT timescaledb_pre_restore();

\! time pg_restore -j 4 -Fc -d database_development /var/lib/postgresql/backups/backup_2020-02-29

SELECT timescaledb_post_restore();

Answer 1

Your database system is I/O bound, as you can see from the %iowait value of 63.62.

Increasing maintenance_work_mem might improve the situation a little, but essentially you need faster storage.