Cassandra 服务启动延迟，并在 system.log 文件中读取 SSTables 时出现警告消息“min_index_interval of 128 is too low”

Question

We found a delay of 2 hrs in starting the Cassandra service with WARN in the system.log file for one table. Please find the warnings in a few of the below servers: WARN [SSTableBatchOpen:5] 2022-08-29 10:01:13,732 IndexSummaryBuilder.java:115 - min_index_interval of 128 is too low for 5511836446 expected keys of avg size 64; using interval of 185 instead

Answer 1

Aaron's answer pointed to the right code: Since you have a LOT of keys in a single SSTable, the default min_index_interval is not efficient anymore and Cassandra recomputes it. This then triggers a rewrite of the index summary during startup, and in this very case it's taking a very long time. Aaron's suggestion of using sstablesplit would be a temporary fix as eventually they'll get compacted again and you'll be back to the same situation.

Changes will have to be made in production to remediate anyway, and changing the min_index_interval seems easy enough as a fix, while really being the only thing to do that won't require deep schema changes to reduce the number of partitions per sstable (or compaction strategy changes which could have hard to predict performance impacts).

Note that changing the min_index_interval will not trigger the rewrite of the sstables straight away. Only newly written sstables will get the new setting, which can be (and should be) forced onto all the sstables using nodetool upgradesstables -a .

On a side note, there seem to be a confusion in the comments between the partition index and secondary indexes (indices?). They are two distinct things and the reported warning message is referring to the partition index summary, not secondary indexes.

Answer 2

It's difficult to discern a question from the above, so I'll assume you're wondering why Cassandra is taking 2 hours to start up.

If you look in the source of Cassandra 3.0, there are some clues given in the IndexSummaryBuilder class. Specifically, the calculations just prior to the warning:

if (maxExpectedEntriesSize > Integer.MAX_VALUE)
{
    // that's a _lot_ of keys, and a very low min index interval
    int effectiveMinInterval = (int) Math.ceil((double)(expectedKeys * expectedEntrySize) / Integer.MAX_VALUE);
    maxExpectedEntries = expectedKeys / effectiveMinInterval;
    maxExpectedEntriesSize = maxExpectedEntries * expectedEntrySize;
    assert maxExpectedEntriesSize <= Integer.MAX_VALUE : maxExpectedEntriesSize;
    logger.warn("min_index_interval of {} is too low for {} expected keys of avg size {}; using interval of {} instead",
                minIndexInterval, expectedKeys, defaultExpectedKeySize, effectiveMinInterval);

The comment about " that's a _lot_ of keys " is a big one, and 5,511,836,446 keys is certainly a lot.

The calculations shown in the method above are driven by the number of keys and sampling intervals for a particular SSTable, to build the Partition Summary into RAM. You can see the Partition Summary on the right side of the diagram showing Cassandra's read path below:

Based on this, I would hypothesize that one particular table's SSTable file(s) is getting too big to handle efficiently. Have a look at the underlying data directory for that table. You may have to split some of those files with tools/bin/sstablesplit to make them more manageable.