[英]Spark scala cassandra save/update
I have a spark dataset of an entity which should get saved/updated into cassandra table named 'offer'.我有一个实体的火花数据集,它应该保存/更新到名为“offer”的 cassandra 表中。
case class Offer(offer_id: String, metadata_last_modified_source_time: java.sql.Timestamp, product_type: String)
val offerDataset: Dataset[Offer] = ....
I want to save or update the above 'offerDataset' to cassandra with the write timestamp to be decided by the field "metadata_last_modified_source_time" of 'offer' entity.我想将上面的“offerDataset”保存或更新为 cassandra,写入时间戳由“offer”实体的“metadata_last_modified_source_time”字段决定。
offerDataset.rdd.saveToCassandra("cassandra_keyspace", "cassandra_table", writeConf = WriteConf(timestamp = TimestampOption.perRow("metadata_last_modified_source_time")))
While writing to Cassandra, am facing the below exception.在写入 Cassandra 时,我面临以下异常。 Can somebody help me to understand this issue.
有人可以帮我理解这个问题。 Got the same error with util.Date and Long types for 'metadata_last_modified_source_time'.
'metadata_last_modified_source_time' 的 util.Date 和 Long 类型出现相同的错误。
com.datastax.driver.core.exceptions.InvalidTypeException: Value metadata_last_modified_source_time is of type bigint, not timestamp
at com.datastax.driver.core.AbstractGettableByIndexData.checkType(AbstractGettableByIndexData.java:83)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:529)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:536)
at com.datastax.driver.core.BoundStatement.set(BoundStatement.java:870)
at com.datastax.spark.connector.writer.BoundStatementBuilder.com$datastax$spark$connector$writer$BoundStatementBuilder$$bindColumnNull(BoundStatementBuilder.scala:59)
at com.datastax.spark.connector.writer.BoundStatementBuilder$$anonfun$5.apply(BoundStatementBuilder.scala:83)
I found solution after going through this doc - https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md我通过这个文档找到了解决方案 - https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md
Introduced a new field writeTime in the Offer case class which should map to cassandra table's write timestamp在Offer案例 class 中引入了一个新字段writeTime ,它应该 map 到 cassandra 表的写入时间戳
case class Offer(offer_id: String, metadata_last_modified_source_time: java.sql.Timestamp, product_type: String, writeTime: sql.Date)
While building the offerDataSet, I set the writeTime field's value to be在构建 offerDataSet 时,我将 writeTime 字段的值设置为
val offerDataset: Dataset[Offer] = {....
....
val writeTime = new Date(metadata_last_modified_source_time.getTime())
....
....
}
offerDataset.rdd.saveToCassandra("cassandra_keyspace", "cassandra_table", writeConf = WriteConf(timestamp = TimestampOption.perRow("writeTime")))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.