Append pandas dataframe 到databricks中的现有表

Question

I want to append a pandas dataframe (8 columns) to an existing table in databricks (12 columns), and fill the other 4 columns that can't be matched with None values. Here is I've tried:

spark_df = spark.createDataFrame(df)
spark_df.write.mode("append").insertInto("my_table")

It thrown the error:

ParseException: "\nmismatched input ':' expecting (line 1, pos 4)\n\n== SQL ==\n my_table

Looks like spark can't handle this operation with unmatched columns, is there any way to achieve what I want?

Answer 1

I think that the most natural course of action would be a select() transformation to add the missing columns to the 8-column dataframe, followed by a unionAll() transformation to merge the two.

from pyspark.sql import Row
from pyspark.sql.functions import lit

bigrow = Row(a='foo', b='bar')
bigdf = spark.createDataFrame([bigrow])
smallrow = Row(a='foobar')
smalldf = spark.createDataFrame([smallrow])

fitdf = smalldf.select(smalldf.a, lit(None).alias('b'))

uniondf = bigdf.unionAll(fitdf)

Answer 2

Can you try this

df = spark.createDataFrame(pandas_df)

df_table_struct = sqlContext.sql('select * from my_table limit 0')

for col in set(df_table_struct.columns) - set(df.columns):
    df = df.withColumn(col, F.lit(None))

df_table_struct = df_table_struct.unionByName(df)

df_table_struct.write.saveAsTable('my_table', mode='append')