removing a node from the cluster and tables in twcs

Question

I have a cluster (tested it both with 2.1.14 and 3.0.17) in which I have a table that is TWCS (time window compaction). All sstables are kept in the correct windows just fine up until I remove a node from the cluster (in the same dc) and in that moment it seems all sstables are treated as one pool for normal size tiered, causing sstables from different time periods to join. Seeing as my cluster is 400 nodes spread over 6 datacenters a node removal is something quite common.

I did not find any bug talking about this, is this expected behavior? having all the sstables handled together causes a major problem space wise since it means new and old data are in the same sstable causing the old data to remain on disk much longer

(2.1 twcs is achieved using a jar from jeffjirsa github)

Answer 1

Have you disabled read repair on TWCS tables? it could inject out-of-sequence timestamps. TWCS itself will do size tiered but only on current window, iff it falls behind in compaction.