Databricks 中的 PySpark SQL：如何从具有相似名称的表中提取名称？

Question

I have the following tables:

Table_A     Table_B    Table_C   List_A

Table_A:

Column_A             Column_B     Column_C     Column_D
1/1/2020             30           400000       Table_A
2/1/2020             35           200000       Table_A

Table_B:

Column_A             Column_B     Column_C     Column_D    
1/1/2020             50           4000         Table_B
2/1/2020             70           10000        Table_B

Table_C:

Column_A             Column_B     Column_C     Column_D
1/1/2020             3            300          Table_C
2/1/2020             5            200          Table_C  

List_A:

Column_A             Line_E                    Column_D       
1/1/2020  09:30:00   30                        List_A
2/1/2020  09:31:00   28                        List_A

I want to select all columns from tables that have similar names. In this example, they are Table_A, Table_B, Table_C . And their names begin with "Table_".

How to do it?

I tried spark.sql("SELECT * FROM * where Column_D like 'Table_*'") , but it does not work.

Answer 1

Tables with similar names - are they tables that:

• all start with Table_ ?
• all start with the same prefix of length 5?
• all start with the same prefix followed by an underscore?

Consider table names like Table_A , Tablerone_B , Table_bable_C , Table_D_D - similar or not...?

With SQL you can get the list of tables using SHOW TABLES , but it is not possible to run it in a subquery, so you are not able to further process it using SQL. But you can use Scala or Python.

spark.sql("show tables").select("tableName").as[String].collect.groupBy(_.split("_")(0))

This returns Map(list -> Array(list_a), table -> Array(table_a, table_b, table_c))

Then you can obtain column names by iterating on above result and running DESCRIBE <table_name> - again you need to think what you want to get (all column names? shared column names?)