Spark scala cassandra 保存/更新

Question

我有一個實體的火花數據集，它應該保存/更新到名為“offer”的 cassandra 表中。

case class Offer(offer_id: String, metadata_last_modified_source_time: java.sql.Timestamp, product_type: String)
val offerDataset: Dataset[Offer] = ....

我想將上面的“offerDataset”保存或更新為 cassandra，寫入時間戳由“offer”實體的“metadata_last_modified_source_time”字段決定。

offerDataset.rdd.saveToCassandra("cassandra_keyspace", "cassandra_table", writeConf = WriteConf(timestamp = TimestampOption.perRow("metadata_last_modified_source_time")))

在寫入 Cassandra 時，我面臨以下異常。 有人可以幫我理解這個問題。 'metadata_last_modified_source_time' 的 util.Date 和 Long 類型出現相同的錯誤。

com.datastax.driver.core.exceptions.InvalidTypeException: Value metadata_last_modified_source_time is of type bigint, not timestamp
at com.datastax.driver.core.AbstractGettableByIndexData.checkType(AbstractGettableByIndexData.java:83)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:529)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:536)
at com.datastax.driver.core.BoundStatement.set(BoundStatement.java:870)
at com.datastax.spark.connector.writer.BoundStatementBuilder.com$datastax$spark$connector$writer$BoundStatementBuilder$$bindColumnNull(BoundStatementBuilder.scala:59)
at com.datastax.spark.connector.writer.BoundStatementBuilder$$anonfun$5.apply(BoundStatementBuilder.scala:83)

Answer 1

我通過這個文檔找到了解決方案 - https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md

在Offer案例 class 中引入了一個新字段writeTime ，它應該 map 到 cassandra 表的寫入時間戳

case class Offer(offer_id: String, metadata_last_modified_source_time: java.sql.Timestamp, product_type: String, writeTime: sql.Date)

在構建 offerDataSet 時，我將 writeTime 字段的值設置為

val offerDataset: Dataset[Offer] = {....
   ....
    val writeTime = new Date(metadata_last_modified_source_time.getTime())
   ....
   ....
}

offerDataset.rdd.saveToCassandra("cassandra_keyspace", "cassandra_table", writeConf = WriteConf(timestamp = TimestampOption.perRow("writeTime")))