Pandas 迭代到 Pyspark Function

Question

Given a dataset I tried to create a logic where within two column I need to enforce continuity in terms of last destination (in to column) being the exact next starting point (in from column) per id. For instance this table

+----+-------+-------+
| id | from  | to    |
+----+-------+-------+
|  1 | A     | B     |
|  1 | C     | A     |
|  2 | D     | D     |
|  2 | F     | G     |
|  2 | F     | F     |
+----+-------+-------+

should ideally look like this:

+----+-------+-------+
| id | from  | to    |
+----+-------+-------+
|  1 | A     | B     |
|  1 | B     | C     |
|  1 | C     | A     |
|  2 | D     | D     |
|  2 | D     | F     |
|  2 | F     | G     |
|  2 | G     | F     |
|  2 | F     | F     |
+----+-------+-------+

Using Pandas I did this by looping row wise and checking if previous_row['to'] == current_row['from'], also a check for id that can be probably avoided using a groupby, as you may see below

for i in range(len(df)):
    if (i < (len(df)-1)):
        if (new.ix[i,"to"] != new.ix[i+1,"from"]) & (new.ix[i,"id"] == new.ix[i+1,"id"]): 
            new_index = i + 0.5
            line = pd.DataFrame({"id":new.ix[i,"id"],
                             "from":new.ix[i,"to"],"to":new.ix[i+1,"from"],}, index = [new_index])
            appendings = pd.concat([appendings,line])
        else:
            pass
    else:
        pass

Is it possible to as is "translate" this to pyspark rdds?

I am aware that looping is far from optimal in Pyspark to replicate a looping and if-else logic.

I considered grouping by and zipping from and to columns and working on a single column. Main problem with this lies in the fact that I could produce a flag on lines that are "faulty" but there is no way to insert new lines without using index-wise operations.

Answer 1

This isn't a pyspark answer but a partial answer to show you how to achieve the task without loop in pandas .

You can try:

def f(sub_df):
    return sub_df.assign(to_=np.roll(sub_df.To, 1)) \
                .apply(lambda x: [[x.From, x.To]] if x.to_ == x.From else [[x.to_, x.From], [x.From, x.To]], axis=1) \
                .explode() \
                .apply(pd.Series)


out = df.groupby('id').apply(f) \
        .reset_index(level=1, drop=True) \
        .rename(columns={0: "from", 1: "to"})

---

Workflow:

• Group the dataframe by id using groupby
• For each group:
  • Create a new column (here names to_ ) to have the previous row. np.roll performs a circular shift in order to keep the last value.
  • According if the current from == previous to : return current line or add a new line to make the transition.
  • Use explode to explode the list of list into one list per row.
  • Convert the column into two columns using apply(pd.Series)
• Then, for the output dataframe, remove the level 1 index using reset_index
• And rename the columns using rename

---

Full code

# Import module
import pandas as pd
import numpy as np

# create dataset
df = pd.DataFrame({"id": [1,1,2,2,2], "From": ["A", "C", "D", "F", "F"], "To": ["B", "D", "D", "G", "F"]})
# print(df)


def f(sub_df):
    return sub_df.assign(to_=np.roll(sub_df.To, 1)) \
                .apply(lambda x: [[x.From, x.To]] if x.to_ == x.From else [[x.to_, x.From], [x.From, x.To]], axis=1) \
                .explode() \
                .apply(pd.Series)


out = df.groupby('id').apply(f) \
        .reset_index(level=1, drop=True) \
        .rename(columns={0: "from", 1: "to"})
print(out)
#    from to
# id
# 1     D  A
# 1     A  B
# 1     B  C
# 1     C  D
# 2     F  D
# 2     D  D
# 2     D  F
# 2     F  G
# 2     G  F
# 2     F  F

The next step is to translate it into PySpark . Try it and feel free to open a new question with yours attempts.