带有rocksdb的Flink增量检查点使用大量内存

Question

Now I'm using incremental Checkpoint in Flink with RocksDB, running on a container environment. As I know, rocksdb will use a lot of memory when doing incremental checkpoint, there is already a JIRA describe this problem: https://issues.apache.org/jira/browse/FLINK-7289 I have tried to adjust my Rocksdb's configuration, but my container still be killed because of OOM. Here are the monitor page: my container will be killed and restart and then killed again.

Here are my configurations:

public class BackendOptions implements OptionsFactory {

@Override
public DBOptions createDBOptions(DBOptions dbOptions) {
    return dbOptions
            .setIncreaseParallelism(4)
            .setUseFsync(false)
            .setMaxOpenFiles(-1);

}
@Override
public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions columnFamilyOptions) {
    return columnFamilyOptions.setCompactionStyle(CompactionStyle.LEVEL)
            .setLevelCompactionDynamicLevelBytes(true)
            .setTargetFileSizeBase(256 * 1024 * 1024)
            .setWriteBufferSize(64 * 1024 * 1024)
            .setMaxBytesForLevelBase(1024 * 1024 * 1024)
            .setMinWriteBufferNumberToMerge(2)
            .setMaxWriteBufferNumber(5)

            .setOptimizeFiltersForHits(true)
            .setTableFormatConfig(
                    new BlockBasedTableConfig()
                            .setBlockCacheSize(256 * 1024 * 1024)  // 256 MB
                            .setBlockSize(128 * 1024) //// 128 KB
                            .setCacheIndexAndFilterBlocks(true)
            );
}

I make my checkpoint every 1 minutes and state size is about 5GB. Can somebody help me or tell me some right way to use incremental checkpoint ?

Answer 1

This appears to be fixed in the newer versions of Flink, being 1.10 and above. The question is about 2019 and per feb 2020 the linked issue got closed.

Details can be found here https://issues.apache.org/jira/browse/FLINK-7289