I have the following code, it will read table from mysql database:
val jdbcDF = sparkSession.read
.format("jdbc")
.option("url", "idbc:mysql location")
.option("dbtable", "tablename")
.option("user", "root")
.option("password", "root")
.load().where(some condition)
My questions:
Unless you will perform some action spark won't execute anything even filter condition and also copying data into memory. It's lazy evaluation.
The code you have written once action will be performed data will be pulled into memory and then filter will be applied. If you want to execute filter in MySQL then pass the query instead of table name in the dbtable option.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.