Spark SQL：是否可以为给定的数据帧分区以预定义的顺序逐行处理？

Question

I am looking to partition my data frame(df) based on a column (SECURITY_ID) and then run df.foreachpartition(customfunction). This is working fine.

Inside each partition, based on a column (RANK) the data has to be ordered. This is working fine.

Now, based on the the order I want to process row after row in sequential order for each partition. For example -

Base dataframe:

+-------------+----------+----------+------
ACCOUNT_NO|SECURITY_ID| QUANTITY|    RANK|
+-------------+----------+----------+------
  32934789|      290X2|   -98763|       3|
  3S534789|      290X2|    45300|       2|
  3FA34789|      290X2|    12763|       1|
  00000019|      290X2|-10177400|       4|
  92115301|      35G71|     8003|       2|
  91615301|      35G71|    -2883|       1|

After partition and order by

+-------------+----------+----------+------
ACCOUNT_NO|SECURITY_ID| QUANTITY|    RANK|
+-------------+----------+----------+------
  3FA34789|      290X2|    12763|       1|  
  3S534789|      290X2|    45300|       2|
  32934789|      290X2|   -98763|       3|
  00000019|      290X2|-10177400|       4|



+-------------+----------+----------+------
ACCOUNT_NO|SECURITY_ID| QUANTITY|    RANK|
+-------------+----------+----------+------
  91615301|      35G71|    -2883|       1|
  92115301|      35G71|     8003|       2|

Let us consider this partition

+-------------+----------+----------+------
ACCOUNT_NO|SECURITY_ID| QUANTITY|    RANK|
+-------------+----------+----------+------
  3FA34789|      290X2|    12763|       1|  
  3S534789|      290X2|    45300|       2|
  32934789|      290X2|   -98763|       3|
  00000019|      290X2|-10177400|       4|

I need to process rows one after the other based on the rank in the increasing order.

This seems to be fine on single node machine. But I see that the processing is getting jumbled when running on a multi-node cluster.

How can I make sure that the order is guaranteed?

Answer 1

Could you please try with coalesce(1) followed by sort(cols:*) operation over the SECURITY_ID partitioned Datafame to get new Datafame/Dataset sorted by the specified columns, all in ascending order.

df.coalesce(1).sort("RANK").foreach(row => process(row))