使用pyspark分区时循环覆盖模式

Question

#Start and End is a range of dates. 
start = date(2019, 1, 20)
end = date(2019, 1, 22)

for single_date in daterange(start, end):
  query = "(SELECT ID, firstname,lastname,date FROM dbo.emp WHERE date = '%s' ) emp_alias" %((single_date).strftime("%Y-%m-%d %H:%M:%S")) 
  df = spark.read.jdbc(url=jdbcUrl, table=query, properties=connectionProperties)
  df.write.format("parquet").mode("ignore").partitionBy("Date").save("/mnt/data/empData.parquet")

I have data for number of days in a table and I need as parquet files partitioned by date. I have to save by day in loop as data is huge and I can't put all the days like years data in one dataframe. I tried on all save modes. In 'Ignore' mode it saves for first day. In 'Overwrite' mode, it saves of last day. In 'append' mode, it adds the data. What I need is, if data is available for that day it should ignore for that day and leave the data what's already there but if data is not available then create in parquet file partitioned by date. Please help.

Answer 1

There is currently no PySpark SaveMode that will allow you to preserve the existing partitions, while inserting the new ones, if you also want to use Hive partitioning (which is what you're asking for, when you call the method partitionBy ). Note that there is the option to do the opposite, which is to overwrite data in some partitions, while preserving the ones for which there is no data in the DataFrame (set the configuration setting "spark.sql.sources.partitionOverwriteMode" to "dynamic" and use SaveMode.Overwrite when writing datasets).

You can still achieve what you want though, by first creating a set of all the already existing partitions. You could do that with PySpark, or using any of the libraries that will allow you to perform listing operations in file systems (like Azure Data Lake Storage Gen2) or key-value stores (like AWS S3). Once you have that list, you use it to filter the new dataset for the data you still want to write. Here's an example with only PySpark:

In [1]: from pyspark.sql.functions import lit
   ...: df = spark.range(3).withColumn("foo", lit("bar"))
   ...: dir = "/tmp/foo"
   ...: df.write.mode("overwrite").partitionBy("id").parquet(dir)  # initial seeding
   ...: ! tree /tmp/foo
   ...: 
   ...: 
/tmp/foo                                                                        
├── id=0
│   └── part-00001-5d14d286-81e1-4eb1-969e-c0d8089712ce.c000.snappy.parquet
├── id=1
│   └── part-00002-5d14d286-81e1-4eb1-969e-c0d8089712ce.c000.snappy.parquet
├── id=2
│   └── part-00003-5d14d286-81e1-4eb1-969e-c0d8089712ce.c000.snappy.parquet
└── _SUCCESS

3 directories, 4 files

In [2]: df2 = spark.range(5).withColumn("foo", lit("baz"))
   ...: existing_partitions = spark.read.parquet(dir).select("id").distinct()
   ...: df3 = df2.join(existing_partitions, "id", how="left_anti")
   ...: df3.write.mode("append").partitionBy("id").parquet(dir)
   ...: spark.read.parquet(dir).orderBy("id").show()
   ...: 
   ...: 
+---+---+                                                                       
|foo| id|
+---+---+
|bar|  0|
|bar|  1|
|bar|  2|
|baz|  3|
|baz|  4|
+---+---+

As you can see, only 2 partitions were added. The ones that were already existing have been preserved.

Now, getting the existing_partitions DataFrame required a read of the data. Spark won't actually read all of the data though, just the partitioning column and the metadata. As mentioned earlier, you could get this data as well using any of the APIs relevant to where your data is stored. In my particular case as well as in yours, seeing as how you're writing to the /mnt folder on Databricks, I could've simply used the built-in Python function os.walk : dirnames = next(os.walk(dir))[1] , and created a DataFrame from that.

By the way, the reason you get the behaviours you've seen is:

1. ignore mode

   In 'Ignore' mode it saves for first day.

   Because you're using a for-loop and the output directory was initially probably non-existing, the first date partition will be written. In all subsequent iterations of the for-loop, the DataFrameWriter object will not write anymore, because it ses there's already some data (one partitione, for the first date) there.

2. overwrite mode

   In 'Overwrite' mode, it saves of last day.

   Actually, it saved a partition in each iteration of the for-loop, but because you're instructing the DataFrameWriter to overwrite, it will remove all previously existing partitions in the directory. So it looks like only the last one was written.

3. append mode

   In 'append' mode, it adds the data This doesn't need further explanation.

One suggestion: there's probably no need to read from the database multiple times (using the for-loop to create multiple different queries and jdbc-connections). You could probably update the query to have WHERE date BETWEEN %(start) AND %(end) , remove the for-loop altogether and enjoy an efficient write.