如何使用spark-cassandra-connector实现LEFT或RIGHT JOIN

Question

I have spark streaming job. I am using Cassandra as datastore. I have stream which is need to be joined with cassandra table. I am using spark-cassandra-connector, there is great method joinWithCassandraTable which is as far as I can understand implementing inner join with cassandra table

val source: DStream[...] = ...
source.foreachRDD { rdd =>
  rdd.joinWithCassandraTable( "keyspace", "table" ).map{ ...
  }
}

So the question is how can I implement left outer join with cassandra table?

Thanks in advance

Answer 1

This is currently not supported, but there is a ticket to introduce the functionality. Please vote on it if you would like it introduced in the future.

https://datastax-oss.atlassian.net/browse/SPARKC-181

A workaround is suggested in the ticket

Answer 2

As RussS mentioned such feature is not available in spark-cassandra-connector driver yet. So as workaround I propose the following code snippet.

rdd.foreachPartition { partition =>
     CassandraConnector(rdd.context.getConf).withSessionDo { session =>
       for (
         leftSide <- partition;
         rightSide <- {
           val rs = session.execute(s"""SELECT * FROM "keyspace".table where id = "${leftSide._2}"""")
           val iterator = new PrefetchingResultSetIterator(rs, 100)
           if (iterator.isEmpty) Seq(None)
           else iterator.map(r => Some(r.getString(1)))
         }
       ) yield (leftSide, rightSide)
     }
   }