如何使用 spark.read.jdbc 读取不同 Pyspark 数据帧中的多个文件

Question

I have a code to read multiple files (>10) into different dataframes in Pyspark. However, I would like to optimize this piece of code using a for loop and a reference variable or something like that. My code is as follows:

Features_PM = (spark.read
          .jdbc(url=jdbcUrl, table='Features_PM',
                properties=connectionProperties))

Features_CM = (spark.read
          .jdbc(url=jdbcUrl, table='Features_CM',
                properties=connectionProperties))

I tried something like this but it didn't work:

table_list = ['table1', 'table2','table3', 'table4']

for table in table_list:
     jdbcDF = spark.read \
         .format("jdbc") \
         .option("url", "jdbc:postgresql:dbserver") \
         .option("dbtable", "schema.{}".format(table)) \
         .option("user", "username") \
         .option("password", "password") \
         .load()

Source for the above snippet: https://community.cloudera.com/t5/Support-Questions/read-multiple-table-parallel-using-Spark/td-p/286498

Any help would be appreciated. Thanks

Answer 1

You can use the following code to achieve your end goal. You will get a dictionary of dataframes where the key is the table name and value is teh appropriate dataframe

def read_table(opts):
    return spark.read.format("jdbc").options(**opts).load()

table_list = ['table1', 'table2','table3', 'table4']



table_df_dict = {table: read_table({"url":"jdbc:postgresql:dbserver",
                                   "dbtable":"schema.{}".format(table),
                                   "user": "username",
                                   "password":"password"})
                 for table in table_list}