Spark on Cassandra：有没有办法按分区键删除数据？

Question

The spark Cassandra connector has the RDD.deleteFromCassandra(keyspaceName, tableName) method.

The values in the RDD are interpreted as Primary Key constraints.

I have a table like that :

CREATE TABLE table (a int, b int, c int, PRIMARY KEY (a,b));

As you can see, a is the partition key , and b the clustering key .

I need to have a spark app that remove efficiently by partition_key , and not by primary key .

Indeed, my goal is to always drop entire partitions by their partition keys , and not create a thombstones for each primary key .

How to do that with spark connector ?

Thank you

Answer 1

Yes, it's possible to do if you specify keyColumns parameter to the .deleteFromCassandra function ( docs ). For example, if you have composite partition key consisting of two columns part1 & part2 :

rdd.deleteFromCassandra("keyspace", "table", 
  keyColumns = SomeColumns("part1", "part2"))

This method works only with RDDs, if you use DataFrames, then you just need to do df.rdd . Also, in some versions of connector, you may need to restrict selection just to partition columns - see discussion in this answer .