Pandas：聚合列切片为 arrays

Question

I have a Pandas dataframe that looks as such:

                      Scaled
               Date 
2020-07-01 02:40:00 0.604511
2020-07-01 02:45:00 0.640577
2020-07-01 02:50:00 0.587683
2020-07-01 02:55:00 0.491515
....

I am trying to add a new column called X which is supposed to look as such, where every two previous values become an array:

                      Scaled   X
               Date 
2020-07-01 02:40:00 0.604511 nan
2020-07-01 02:45:00 0.640577 nan
2020-07-01 02:50:00 0.587683 [0.604511 0.640577]
2020-07-01 02:55:00 0.491515 [0.640577 0.587683]
...

I am trying a for -loop to do so, but I don't think this is the most elegant and efficient way, so any suggestion of how to do this in pandas? (but it is not going as intended)

window_size = 2
for i in range(window_size, df.shape[0]):
    df['X'][i] = df['Scaled'][i - window_size:window_size] 

Answer 1

To use pandas, you may use list comprehension and concat and shift

window_size = 2
df['X'] = (pd.concat([df.Scaled.shift(-i) for i in range(window_size)], axis=1)
             .shift(window_size).values.tolist())

Out[213]:
     Scaled                               X
0  0.604511                      [nan, nan]
1  0.640577                      [nan, nan]
2  0.587683  [0.604511, 0.6405770000000001]
3  0.491515  [0.6405770000000001, 0.587683]

Answer 2

You had the right idea with using a for loop.

First you have to initialize the new column you can use .apply() on your dataframe to do this.

Then you can use .iterrows() to iterate through the indices of the dataframe creating your desired array as you iterate through the rows.

import pandas as pd

df = pd.DataFrame(data={'Date': ['2020-07-01 02:40:00', '2020-07-01 02:45:00', '2020-07-01 02:50:00', '2020-07-01 02:55:00'], 'Scaled': [0.604511, 0.640577, 0.587683, 0.491515]})

df['New_col'] = df['Scaled'].apply(lambda x : float("NAN"))

for i, val in df.iterrows():
  if i == 0 or i == 1:
    scaled_a = None
    scaled_b = None
  else:
    scaled_a = df['Scaled'][i-2]
    scaled_b = df['Scaled'][i-1]
  df['New_col'][i] = [scaled_a, scaled_b] 

Simply assign the value of your new column to the value of the dataframe Scaled column at the previous two indices an save it in an array. Hope this helped!!

    Date                Scaled      New_col
0   2020-07-01 02:40:00 0.604511    [None, None]
1   2020-07-01 02:45:00 0.640577    [None, None]
2   2020-07-01 02:50:00 0.587683    [0.604511, 0.640577]
3   2020-07-01 02:55:00 0.491515    [0.640577, 0.587683]

The result should look something like this. ^^

Answer 3

updated Identical output. This is a pandas implementation. Uses numpy generate list that is pandas column for df that is super efficient.

d = list(pd.date_range(dt.datetime(2020,7,1), dt.datetime(2020,7,2), freq="15min"))
df = pd.DataFrame({"Date":d, 
      "Scaled":[round(Decimal(random.uniform(0, 1)),6) for x in d]})


# generate two new arrays that are shifted version of *scaled*
a1 = np.roll(df["Scaled"],1)
a1[0:2] = None
a2 = np.roll(df["Scaled"],2)
a2[0:2] = None
# combine them into a list and put back into df
df['X'] = np.vstack((a2, a1)).T.tolist()

print(df[:10].to_string(index=False))

output

               Date    Scaled                     X
2020-07-01 00:00:00  0.396534          [None, None]
2020-07-01 00:15:00  0.890777          [None, None]
2020-07-01 00:30:00  0.241534  [0.396534, 0.890777]
2020-07-01 00:45:00  0.800615  [0.890777, 0.241534]
2020-07-01 01:00:00  0.161382  [0.241534, 0.800615]
2020-07-01 01:15:00  0.727410  [0.800615, 0.161382]
2020-07-01 01:30:00  0.146833  [0.161382, 0.727410]
2020-07-01 01:45:00  0.925441  [0.727410, 0.146833]
2020-07-01 02:00:00  0.770211  [0.146833, 0.925441]
2020-07-01 02:15:00  0.310082  [0.925441, 0.770211]

Answer 4

Here's a version without the for loop. First, create the data frame:

from io import StringIO

data = '''Date  Scaled 
2020-07-01 02:40:00  0.604511
2020-07-01 02:45:00  0.640577
2020-07-01 02:50:00  0.587683
2020-07-01 02:55:00  0.491515
'''
df = pd.read_csv(StringIO(data), sep='\s\s', engine='python')

Next, use shift() to get previous values, and the lambda function creates the 2-element list or yields a single NaN:

f = lambda a, b: np.nan if np.isnan(a) or np.isnan(b) else [a, b]

window_size = 2

t = (pd.concat([df['Scaled'].shift(window_size).rename('a'), 
                df['Scaled'].shift(window_size - 1).rename('b')], axis=1
          )
       .apply(lambda x: f(x['a'].round(6), x['b'].round(6)), axis=1))

print(t)

0                     NaN
1                     NaN
2    [0.604511, 0.640577]
3    [0.640577, 0.587683]
dtype: object