Spark Streaming 数据帧持久化操作

Question

I am reading Oracle database from my spark code and I persist it - (cache operation).

  val dataOracle = spark.read
  .format("jdbc")
  .option("url",conn_url)
  .option("dbtable", s"(select * from table)")
  .option("user", oracle_user)
  .option("password", oracle_pass)
  .option("driver",oracle_driver)
  .load().persist()

End of the code, I need unpersist this dataframe, cause it can be happened some changes in database and I need those data in the next cycle, but at the same time time cost so important to me. If I cache the dataframe my code takes under the 1 second, if I dont above 3 second(which is not acceptable). Is there any strategy to get latest data from DB, also minimized time cost value!

There is the my main operation using Oracle data:

dataOracle.createOrReplaceTempView("TABLE")
val total = spark.sql(s"select count(*) from TABLE where name = ${name}").first().getLong(0)
val items = spark.sql(s"SELECT count(*) from TABLE where index = ${id} and name = ${name}").first().getLong(0)
val first_rule: Double = total.toDouble / items.toDouble

Answer 1

If your dataframe is updated and you need those updates, then by definition you can't cache anything and you just need to read it all over again. A possible way to optimize is to add a column of last modified timestamp to your table in the database and only read those entries where the last modified timestamp is greater than some value.