如何减少 postgresql 中生成的 WAL 文件数量
[英]how to reduce amount of WAL files generated in postgresql
问题描述
Huge pile of WAL files are generated in Master-Standby replication. 
Master-Standby 复制产生了一大堆WAL 文件。 walfiles are archived at one of the standby node and every 2 hour, we are using tar to compress the archived WALs in standby node. walfiles 在备用节点之一存档，每 2 小时，我们使用tar压缩备用节点中的存档 WAL。 Still, it becomes a huge size to store.尽管如此，它的尺寸仍然很大。 When it comes to 30, 90 days' backup it becomes a huge storage issue.当谈到 30 天、90 天的备份时，它就变成了一个巨大的存储问题。 Also, ends up taking more time to download and replay the WAL's during restoration.此外，最终会花费更多时间在恢复期间下载和重放 WAL。
I have used the below options.我使用了以下选项。
wal_level=replica
wal_compression=on
archive_mode = always
And below parameters are commented/not used.以下参数已注释/未使用。
archive_timeout
checkpoint_timeout
Is there any other way, we can reduce the number of WAL's generated or an easier way to manage them?有没有其他方法，我们可以减少生成的 WAL 的数量或更简单的方法来管理它们？ pg_waldump is showing around 70-90% of the data is full page images. pg_waldump显示大约 70-90% 的数据是整页图像。
Also, Can I make above parameters in effect by changing in standby node?另外，是否可以通过更改备用节点使上述参数生效？ Is standby archiving the same WAL's sent by the master?备用数据库是否归档了主服务器发送的相同 WAL？ OR is it regenerating based on standby's configuration?或者它是根据备用配置重新生成的吗？
-- Update: Modified to below values -- 更新：修改为以下值
name | setting | unit
--------------------+---------+------
archive_timeout | 0 | s
checkpoint_timeout | 3600 | s
checkpoint_warning | 3600 | s
max_wal_size | 4000 | MB
min_wal_size | 2000 | MB
shared_buffers | 458752 | 8kB
wal_buffers | 4096 | 8kB
wal_compression | on |
wal_level | replica |
still seeing 3-4 WAL files generated every minute.仍然看到每分钟生成 3-4 个 WAL 文件。 I am making these changes on hot standby node(From where backup is taken).我正在热备用节点（从那里进行备份）上进行这些更改。 Should I change this in Master?我应该在 Master 中更改它吗？ Does master settings have affect on Standby's WAL generation? master 设置是否影响 Standby 的 WAL 生成？
Example pg_waldump showing FPI size=87%示例 pg_waldump 显示 FPI 大小=87%
pg_waldump --stats 0000000100000498000000B2
Type N (%) Record size (%) FPI size (%) Combined size (%)
---- - --- ----------- --- -------- --- ------------- ---
XLOG 1 ( 0.00) 114 ( 0.01) 0 ( 0.00) 114 ( 0.00)
Transaction 3070 ( 10.35) 104380 ( 4.86) 0 ( 0.00) 104380 ( 0.63)
Storage 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
CLOG 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Database 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Tablespace 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
MultiXact 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
RelMap 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Standby 2 ( 0.01) 100 ( 0.00) 0 ( 0.00) 100 ( 0.00)
Heap2 590 ( 1.99) 33863 ( 1.58) 46192 ( 0.32) 80055 ( 0.48)
Heap 6679 ( 22.51) 578232 ( 26.92) 4482508 ( 30.92) 5060740 ( 30.41)
Btree 19330 ( 65.14) 1430918 ( 66.62) 9967524 ( 68.76) 11398442 ( 68.48)
Hash 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Gin 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Gist 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Sequence 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
SPGist 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
BRIN 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
CommitTs 4 ( 0.01) 120 ( 0.01) 0 ( 0.00) 120 ( 0.00)
ReplicationOrigin 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
Generic 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
LogicalMessage 0 ( 0.00) 0 ( 0.00) 0 ( 0.00) 0 ( 0.00)
-------- -------- -------- --------
Total 29676 2147727 [12.90%] 14496224 [87.10%] 16643951 [100%]
After using log_checkpoints=on使用log_checkpoints=on后
2022-06-15 07:08:57 UTC [11] LOG: checkpoint starting: time
2022-06-15 07:29:57 UTC [11] LOG: checkpoint complete: wrote 67010 buffers (14.6%); 0 WAL file(s) added, 12 removed, 56 recycled; write=1259.767 s, sync=0.010 s, total=1259.961 s; sync files=253, longest=0.003 s, average=0.001 s; distance=1125728 kB, estimate=2176006 kB
2022-06-15 07:38:57 UTC [11] LOG: checkpoint starting: time
2022-06-15 07:59:57 UTC [11] LOG: checkpoint complete: wrote 61886 buffers (13.5%); 0 WAL file(s) added, 20 removed, 10 recycled; write=1259.740 s, sync=0.005 s, total=1259.878 s; sync files=185, longest=0.002 s, average=0.001 s; distance=491822 kB, estimate=2007588 kB
3 个解决方案
解决方案1
2 2022-06-10 04:13:34
wal_compression=on
This may be counter-productive.这可能会适得其反。 This type of compression needs to compress each WAL record in isolation, without the larger context.这种类型的压缩需要在没有更大上下文的情况下单独压缩每个 WAL 记录。 So this is not very effective.所以这不是很有效。 However, when you then recompress whole WAL files offline where they do have access to the larger context, the first round of attempted compression interferes with the better-situated compression attempt.但是，当您随后离线重新压缩整个 WAL 文件时，它们确实可以访问更大的上下文，第一轮尝试的压缩会干扰位置更好的压缩尝试。
For example, if I take the WAL from 1,000,000 pgbench transactions, they occupy 889192448 raw bytes without wal_compression, and 637534208 with it.例如，如果我从 1,000,000 个 pgbench 事务中获取 WAL，它们在没有 wal_compression 的情况下占用 889192448 个原始字节，在有 wal_compression 的情况下占用 637534208 个字节。
But then after passing them through 'xz' (a very slow but very thorough compressor), the first set takes 129393020 bytes but the 2nd one takes 155769400. So turning on compression too soon cost me 20% more space.但是在将它们通过“xz”（一个非常慢但非常彻底的压缩器）后，第一组占用 129393020 字节，而第二组占用 155769400 字节。因此过早打开压缩使我多花费了 20% 的空间。
You could use pg_waldump --stat... on some WAL files to see what is actually in them.您可以在某些 WAL 文件上使用pg_waldump --stat...来查看其中的实际内容。 If it is mostly FPI, then you could try to make the checkpoints further apart to reduce the FPI frequency.如果它主要是 FPI，那么您可以尝试将检查点分开以降低 FPI 频率。 But if you don't have much FPI to start with, that would be ineffective.但是，如果您一开始没有太多 FPI，那将是无效的。 If you can isolate what is causing so much WAL maybe you can do something about it.如果你能找出导致这么多 WAL 的原因，也许你可以做点什么。 For example if you do a lot of degenerate updates where a column is set to the same value it already had, adding a WHERE to suppress those cases could spare you a lot of WAL generation.例如，如果你做了很多退化更新，其中一列被设置为它已经拥有的相同值，添加一个 WHERE 来抑制这些情况可以节省你很多 WAL 生成。
解决方案2
1 2022-06-09 19:41:32
WALs being generated are a reflection of your primary machine activity.生成的 WAL 反映了您的主要机器活动。 Increasing checkpoint_timeout will help reduce your overall machine activity making it easier to process the WAL logs.增加 checkpoint_timeout 将有助于减少您的整体机器活动，从而更容易处理 WAL 日志。
Standby Archiving is the processing the logs as sent by the Primary.备用存档是处理主服务器发送的日志。 They are binary identical.它们是二进制相同的。 Is it a cold standby or are you processing logs on the standby as they are sent?它是冷备用还是您在发送日志时在备用服务器上处理日志？
解决方案3
0 2022-06-10 06:08:18
Since a high percentage of your WAL consists of full page images, you can reduce the amount of WAL considerably by having checkpoints less often.由于 WAL 的很大一部分由整页图像组成，因此您可以通过减少检查点的频率来显着减少 WAL 的数量。 A full page image is written to WAL whenever a page becomes dirty for the first time after a checkpoint.每当页面在检查点后第一次变脏时，就会将整页图像写入 WAL。 The price you have to pay is a longer crash recovery time.您必须付出的代价是更长的崩溃恢复时间。
To reduce the rate of checkpoints, change these parameters:要降低检查点的速率，请更改以下参数：
checkpoint_timeout(default 5 minutes): set it to something high like 1 hourcheckpoint_timeout（默认 5 分钟）：将其设置为 1 小时之类的高值max_wal_size(default 1GB): set it higher than the amount of WAL that is written withing one hour to match thecheckpoint_timeoutsettingmax_wal_size（默认 1GB）：将其设置为高于一小时内写入的 WAL 数量以匹配checkpoint_timeout设置
These settings have to be made on the primary server, where WAL is generated, not on the standby.这些设置必须在生成 WAL 的主服务器上进行，而不是在备用服务器上进行。 Best practice is to use the same settings on both servers.最佳做法是在两台服务器上使用相同的设置。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.