交易区块 | Spark SQL, rdd

Question

I have a processing block inside my dStream.foreachRDD method and that processing includes persisting to mysql using spark sql. Post that, I am persisting the latest processed offset in another schema/table. I want to make the entire block transactional(scala). How to achieve that? Following are the relevant excerpts from the code :

foreachRDD(rdd => {
  ...........
  ...................................


  df.write.mode("append") .jdbc(url + rawstore_schema +"?rewriteBatchedStatements=true",tablesToFetch(index),connectionProperties)

  ....................
  metricsStatement.executeUpdate("Insert into metrics.txn_offsets (topic,part,off,date_updated) values (...........................

}

as both the write operations(processed data and offset data) are done on two different database/connections, how to make them transactional?

Thanks

Answer 1

I had the same question. Looking through the Spark code (up to v2.1) it doesn't seem to be possible, there's no option to specify transaction management.

More details in my other answer here: https://stackoverflow.com/a/42964361/47551