Scala/Spark 判断外部表的路径

Question

I am having one external table on on gs bucket and to do some compaction logic, I want to determine the full path on which the table is created.

val tableName="stock_ticks_cow_part"
val primaryKey="key"
val versionPartition="version"
val datePartition="dt"
val datePartitionCol=new org.apache.spark.sql.ColumnName(datePartition)

import spark.implicits._

val compactionTable = spark.table(tableName).withColumnRenamed(versionPartition, "compaction_version").withColumnRenamed(datePartition, "date_key")
compactionTable. <code for determining the path>

Let me know if anyone knows how to determine the table path in scala.

Answer 1

I think you can use .inputFiles to

Returns a best-effort snapshot of the files that compose this Dataset

Be aware that this returns an Array[String] , so you should loop through it to get all information you're looking for.

So actually just call

compactionTable.inputFiles

and look at each element of the Array

Answer 2

Here is the correct answer:

import org.apache.spark.sql.catalyst.TableIdentifier
lazy val tblMetadata = catalog.getTableMetadata(new TableIdentifier(tableName,Some(schema)))

lazy val s3location: String = tblMetadata.location.getPath

Answer 3

You can use SQL commands SHOW CREATE TABLE <tablename> or DESCRIBE FORMATTED <tablename> . Both should return the location of the external table, but they need some logic to extract this path...

See also How to get the value of the location for a Hive table using a Spark object?

Answer 4

Use the DESCRIBE FORMATTED SQL command and collect the path back to the driver.

In Scala:

val location = spark.sql("DESCRIBE FORMATTED table_name").filter("col_name = 'Location'").select("data_type").head().getString(0)

The same in Python:

location = spark.sql("DESCRIBE FORMATTED table_name").filter("col_name = 'Location'").select("data_type").head()[0]