![](/img/trans.png)
[英]Spark 3.2.0 Structured Streaming save data to Kafka with Confluent Schema Registry
[英]Spark Structured Streaming from kafka to save data in Cassandra in Distributed fashion
我正在嘗試創建從Kafka到Spark的結構化流,這是一個json字符串。 現在想將json解析為特定的列,然后以最佳速度將數據幀保存到cassandra表中。 使用Spark 2.4和cassandra 2.11(Apache)而非DSE。
我嘗試創建一個直接流,該流提供了案例類的DStream,我在DStream上使用foreachRDD將其保存到Cassandra中,但是每隔6-7天就會掛起。 因此,嘗試流式處理直接提供數據幀並可以將其保存到Cassandra。
val conf = new SparkConf()
.setMaster("local[3]")
.setAppName("Fleet Live Data")
.set("spark.cassandra.connection.host", "ip")
.set("spark.cassandra.connection.keep_alive_ms", "20000")
.set("spark.cassandra.auth.username", "user")
.set("spark.cassandra.auth.password", "pass")
.set("spark.streaming.stopGracefullyOnShutdown", "true")
.set("spark.executor.memory", "2g")
.set("spark.driver.memory", "2g")
.set("spark.submit.deployMode", "cluster")
.set("spark.executor.instances", "4")
.set("spark.executor.cores", "2")
.set("spark.cores.max", "9")
.set("spark.driver.cores", "9")
.set("spark.speculation", "true")
.set("spark.locality.wait", "2s")
val spark = SparkSession
.builder
.appName("Fleet Live Data")
.config(conf)
.getOrCreate()
println("Spark Session Config Done")
val sc = SparkContext.getOrCreate(conf)
sc.setLogLevel("ERROR")
val ssc = new StreamingContext(sc, Seconds(10))
val sqlContext = new SQLContext(sc)
val topics = Map("livefleet" -> 1)
import spark.implicits._
implicit val formats = DefaultFormats
val df = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "brokerIP:port")
.option("subscribe", "livefleet")
.load()
val collection = df.selectExpr("CAST(value AS STRING)").map(f => parse(f.toString()).extract[liveevent])
val query = collection.writeStream
.option("checkpointLocation", "/tmp/check_point/")
.format("kafka")
.format("org.apache.spark.sql.cassandra")
.option("keyspace", "trackfleet_db")
.option("table", "locationinfotemp1")
.outputMode(OutputMode.Update)
.start()
query.awaitTermination()
預期是將數據幀保存到cassandra。 但是得到這個錯誤:-
線程“主”中的異常org.apache.spark.sql.AnalysisException:具有流源的查詢必須使用writeStream.start()執行
根據錯誤消息,我會說Cassandra不是Streaming Sink,並且我相信您需要使用.write
collection.write
.format("org.apache.spark.sql.cassandra")
.options(...)
.save()
要么
import org.apache.spark.sql.cassandra._
// ...
collection.cassandraFormat(table, keyspace).save()
文件: https : //github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md#example-using-helper-commands-to-write-datasets
但這可能僅適用於數據幀,流源,請參見此示例 , 該示例使用.saveToCassandra
import com.datastax.spark.connector.streaming._
// ...
val wc = stream.flatMap(_.split("\\s+"))
.map(x => (x, 1))
.reduceByKey(_ + _)
.saveToCassandra("streaming_test", "words", SomeColumns("word", "count"))
ssc.start()
如果那行不通,那么您確實需要一個ForEachWriter
collection.writeStream
.foreach(new ForeachWriter[Row] {
override def process(row: Row): Unit = {
println(s"Processing ${row}")
}
override def close(errorOrNull: Throwable): Unit = {}
override def open(partitionId: Long, version: Long): Boolean = {
true
}
})
.start()
同樣值得一提的是,Datastax發布了Kafka連接器,並且Kafka Connect隨您的Kafka安裝(假定為0.10.2)或更高版本一起提供。 你可以在這里找到它的公告
如果您使用的是Spark 2.4.0,請嘗試使用foreachbatch編寫器。 它在流查詢中使用基於批處理的編寫器。
val query= test.writeStream
.foreachBatch((batchDF, batchId) =>
batchDF.write
.format("org.apache.spark.sql.cassandra")
.mode(saveMode)
.options(Map("keyspace" -> keySpace, "table" -> tableName))
.save())
.trigger(Trigger.ProcessingTime(3000))
.option("checkpointLocation", /checkpointing")
.start
query.awaitTermination()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.