Table in cassandra stops responding

Question

I have a table in cassandra with the following table structure.

CREATE TABLE "TagIdKeySpace"."TagReadingsFailed" (
"ServerUrl" text,
"TagId" text,
"FromTime" timestamp,
"ToTime" timestamp,
"AvgReading" decimal,
"InsertTimeStamp" timestamp,
"MaxReading" decimal,
"MinReading" decimal,
"Readings" text,
PRIMARY KEY (( "ServerUrl", "TagId" ), "FromTime", "ToTime")
) WITH bloom_filter_fp_chance = 0.01
AND comment = ''
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE'
AND caching = {
    'keys' : 'ALL',
'rows_per_partition' : 'NONE'
    }
      AND compression = {
'chunk_length_in_kb' : 64,
'class' : 'LZ4Compressor',
'enabled' : true
}
AND compaction = {
'class' : 'SizeTieredCompactionStrategy',
'max_threshold' : 32,
'min_threshold' : 4
 };

And I have a similar table in another Cassandra Database in a different machine.

What I want to do is transfer the records from this data base to a central database, after the transfer is complete I would like to delete the records in the local database

For that I have used CassandraCsharpDriver (in ac# windows service application)to select records from the local table, make a web request to send the data to the other machine and then delete the records that are present in the local machine.

For a while it worked fine.

Suddenly I noticed that when I run the following query,

select distinct ("ServerUrl","TagId") from "TagReadingsFailed"  I get an Error: Key not found in dictionary.

When I use The Datastax devcenter to run the same query. It shows the Error: ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures"

info={'failures': 1, 'received_responses': 0, 'required_response': 1,'conistency': 'ONE'}

I have tried running the nodetool repair but it says that nothing to be repaired in the keyspace.

If I run a select query that has the both the clustring and the partitioning keys then the query executes without any errors in this table. If I run a query that selects count, distinct, * (select * from "TagReadingsFailed") It shows the above error.

I would like to know what it is that I am doing wrong

Answer 1

I am guessing the issue is with tombstone. There is huge number of tombstone while removing records for this table. Please refer to this for more details. Another easier way is to drop the table and create again, as you already removed all records. The second approach is to wait for compaction process to trigger (you can change related compaction parameters to make it run more frequently).