Spark sql query optimization

Question

I want to load a data table in spark dataframe. i have 2 table in my database.Is it necessary to write 2 time the full connection options? Is there any way to write the common parts once then just change the variable table names multiple time.

table1 = spark.read\
.format("jdbc")\
.option("url","jdbc:oracle:thin:USER/Password@host:port/db_name")\
.option("driver","oracle.jdbc.driver.OracleDriver" )\
.option("dbtable","table_name_1")\
.load()


table2 = spark.read\
    .format("jdbc")\
    .option("url","jdbc:oracle:thin:USER/Password@host:port/db_name")\
    .option("driver","oracle.jdbc.driver.OracleDriver" )\
    .option("dbtable","table_name_2")\
    .load()

Answer 1

Please below snippet, hope it helps you.

def load_table_df(table_name):
    # You can define "jdbc:oracle:thin:USER/Password@host:port/db_name" as parameter too.
    return spark.read\
        .format("jdbc")\
        .option("url","jdbc:oracle:thin:USER/Password@host:port/db_name")\
        .option("driver","oracle.jdbc.driver.OracleDriver" )\
        .option("dbtable", table_name)\
        .load()

table1 = load_table_df('table_name_1')
table2 = load_table_df('table_name_2')

Answer 2

You can separate reader creation

reader = (spark.read
  .format("jdbc")
  .option("url","jdbc:oracle:thin:USER/Password@host:port/db_name")
  .option("driver","oracle.jdbc.driver.OracleDriver" ))

and load

table1 = reader.option("dbtable","table_name_1").load()
table2 = reader.option("dbtable","table_name_2").load()