从卡夫卡读到火花的数据在注册成表后会消失吗？

Question

Consider data written from a dataframe to kafka and then read from kafka back out to a new dataframe :

// Write from df to kafka
val wdf  = airj.write
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("topic", "air2008")
  .save

Now read the data back

// Read from kafka into spark df
import org.apache.spark.sql.functions._
val flights = (spark.read
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "air2008")
  .load())

How many records?

scala> flights.count
res36: Long = 5824436

Let's register this as a table:

flights.createOrReplaceTempView("flights_raw")

Let's ask that a different way : how many records .. ??!

spark.sql("select count(1) from flights_raw").show
+--------+
|count(1)|
+--------+
|0       |
+--------+

Let's ask the question the first way again:

scala> flights.count
res40: Long = 0

What happened here ?

Answer 1

Based on a comment from @GiorgosMyrianthous I put a _cache_ in. It only helps if done before the createOrReplaceTempView : as follows

Does not work:

import org.apache.spark.sql.functions._
val flights = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "air2008")
  .load()
flights.createOrReplaceTempView("flights_raw").cache

works :

import org.apache.spark.sql.functions._
val flights = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "air2008")
  .load()
flights.cache
flights.createOrReplaceTempView("flights_raw")

Now it works

scala> flights.count
res47: Long = 5824436

Answer 2

createOrReplaceTempView is lazily evaluated meaning that it does not persist to memory. To do so, you'd have to cache the data.

flights.cache
flights.createOrReplaceTempView("flights_raw")

or

flights.createOrReplaceTempView("flights_raw")
spark.table("flights_raw")
spark.table("flights_raw").cache
spark.table("flights_raw").count

should do the trick.