[英]Improve PostgreSQL pg_restore Performance from 130 hours
I am trying to improve the time taken to restore a PostgreSQL database backup using pg_restore
.我正在尝试缩短使用pg_restore
恢复 PostgreSQL 数据库备份所需的时间。 The 29 GB gzip-compressed backup file is created from a 380 GB PostgreSQL database using pg_dump -Z0 -Fc
piped into pigz
. 29 GB gzip 压缩备份文件是使用pg_dump -Z0 -Fc
从 380 GB PostgreSQL 数据库创建的,通过管道传输到pigz
。
During pg_restore
, the database size is increasing at a rate of 50 MB/minute estimated using the SELECT pg_size_pretty(pg_database_size())
query.在pg_restore
期间,数据库大小以使用SELECT pg_size_pretty(pg_database_size())
查询估计的 50 MB/分钟的速度增加。 At this rate, it will take approximately 130 hours to complete the restore which is a very long time.按照这个速度,完成恢复大约需要 130 个小时,这是一个很长的时间。
On further investigation, it appears that the CPU usage is low despite setting pg_restore
to use 4 workers.在进一步调查中,尽管将pg_restore
设置为使用 4 个工人,但 CPU 使用率似乎很低。
The disk write speed and IOPS are also very low:磁盘写入速度和IOPS也很低:
Benchmarking the system's IO using fio
has shown that it can do 300 MB/s writes and 2000 IOPS, so we are utilizing only about 20% of the potential IO capabilities.使用fio
对系统的 IO 进行基准测试表明,它可以执行 300 MB/s 的写入和 2000 IOPS,因此我们仅利用了大约 20% 的潜在 IO 功能。
Is there any way to speed up the database restore?有什么办法可以加快数据库恢复?
System系统
Steps taken to perform restore:执行恢复所采取的步骤:
Decompress the .gz file to /var/lib/postgresql/backups/backup_2020-02-29
(~ 40mins)将 .gz 文件解压到/var/lib/postgresql/backups/backup_2020-02-29
(约 40 分钟)
Modify postgresql.conf
settings修改postgresql.conf
设置
work_mem = 32MB
shared_buffers = 1GB
maintenance_work_mem = 1GB
full_page_writes = off
autovacuum = off
wal_buffers = -1
pg_ctl restart
Run the following commands inside psql
:在psql
运行以下命令:
CREATE DATABASE database_development;
\c database_development
CREATE EXTENSION timescaledb;
SELECT timescaledb_pre_restore();
\! time pg_restore -j 4 -Fc -d database_development /var/lib/postgresql/backups/backup_2020-02-29
SELECT timescaledb_post_restore();
Your database system is I/O bound, as you can see from the %iowait
value of 63.62.您的数据库系统受 I/O 限制,正如您从%iowait
值 63.62 中看到的那样。
Increasing maintenance_work_mem
might improve the situation a little, but essentially you need faster storage.增加maintenance_work_mem
可能会稍微改善这种情况,但本质上您需要更快的存储。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.