[英]Spark: Create temporary table by executing sql query on temporary tables
我正在使用 Spark,我想知道:如何通過對表 A 和 B 執行 sql 查詢來創建名為 C 的臨時表?
sqlContext
.read.json(file_name_A)
.createOrReplaceTempView("A")
sqlContext
.read.json(file_name_B)
.createOrReplaceTempView("B")
val tableQuery = "(SELECT A.id, B.name FROM A INNER JOIN B ON A.id = B.fk_id) C"
sqlContext.read
.format(SQLUtils.FORMAT_JDBC)
.options(SQLUtils.CONFIG())
.option("dbtable", tableQuery)
.load()
您需要將結果保存為臨時表
tableQuery .createOrReplaceTempView("dbtable")
可以使用 JDBC 的外部表上的永久存儲
val prop = new java.util.Properties
prop.setProperty("driver", "com.mysql.jdbc.Driver")
prop.setProperty("user", "vaquar")
prop.setProperty("password", "khan")
//jdbc mysql url - destination database is named "temp"
val url = "jdbc:mysql://localhost:3306/temp"
//destination database table
val dbtable = "sample_data_table"
//write data from spark dataframe to database
df.write.mode("append").jdbc(url, dbtable, prop)
https://docs.databricks.com/spark/latest/data-sources/sql-databases.html
http://spark.apache.org/docs/latest/sql-programming-guide.html# Saving-to-persistent-tables
sqlContext.read.json(file_name_A).createOrReplaceTempView("A")
sqlContext.read.json(file_name_B).createOrReplaceTempView("B")
val tableQuery = "(SELECT A.id, B.name FROM A INNER JOIN B ON A.id = B.fk_id) C"
sqlContext.sql(tableQuery).createOrReplaceTempView("C")
試試上面的代碼它會起作用。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.