如何在 Pyspark 中处理包含 SQL 逻辑的 Table.column

Question

Eg:

Table - MappingTable

Col1	Col2	MappingLogic
One	Two	SELECT * FROM TableX
One	Two	SELECT * FROM TableX X Left Outer Join TableY Y on X.id=Y.ID

Other Tables - TableX and TableY

How Can I use this mapping table in Pyspark dataframe and build my logic using MappingLogic column??

Answer 1

Not sure what kind of answer are you expecting, but in general you can use sql expressions in your pyspark code. You just have to create views on your tables first:

spark.read \
    .jdbc("jdbc:postgresql:dbserver", "tableX",
          properties={"user": "username", "password": "password"}).createOrReplaceTempView("tableX")

# Later you get sql-expression from your mapping logic table and execute it:
s = "SELECT * FROM TableX"
df = spark.sql(s)