Pandas 相当于 dplyr 点

Question

I am sorry for pretty heavy explanation, but hope you will get the idea.

I'm R user and I find tidyverse capabilities in data wrangling really powerful. But recently I have started learning Python, and in particular pandas to extend my opportunities in data analysis. Instinctively I'm trying to do things in pandas as I used to do them while I was using dplyr.

So my question is whether any equivalent to dplyr dot while you are using method chaining in pandas.

Here example illustrates computing of minimum value from all values that are greater than current value in test_df['data'] per each group and than the same computing but across new column.

R's Example:

require(dplyr)
require(purrr)
test_df = data.frame(group = rep(c(1,2,3), each = 3),
                     data= c(1:9))
test_df %>%
group_by(group) %>%
mutate(., min_of_max = map_dbl(data, ~data[data > .x] %>% min())) %>%
mutate(., min_of_max_2 = map_dbl(min_of_max, ~min_of_max[min_of_max > .x] %>% min()))

Output:

# A tibble: 9 x 4
# Groups:   group [3]
group  data min_of_max min_of_max_2
<dbl> <int>      <dbl>        <dbl>
1     1     1          2            3
2     1     2          3          Inf
3     1     3        Inf          Inf
4     2     4          5            6
5     2     5          6          Inf
6     2     6        Inf          Inf
7     3     7          8            9
8     3     8          9          Inf
9     3     9        Inf          Inf

I know that dplyr doesn't even require dot, but I put it for better understanding the specific of my question

Doing the same in Pandas

Invalid Example:

import pandas as pd
import numpy as np
test_df = (
    pd.DataFrame({'A': np.array([1,2,3]*3), 'B': np.array(range(1,10))})
    .sort_values(by = ['A', 'B'])
)
(test_df.assign(min_of_max = test_df.apply(lambda x: (test_df.B[(test_df.B > x.B) &
                                                           (test_df.A[test_df.A == x.A])]).min(), axis = 1))
    .assign(min_of_max2 = 'assume_dot_here'.apply(lambda x: (test_df.min_of_max[(test_df.min_of_max > x.min_of_max) &
                                                           (test_df.A[test_df.A == x.A])]).min(), axis = 1)))

In this example putting dot in a second .assign would be great ability but it doesn't work in pandas.

Valid Example, which ruins chain:

test_df = test_df.assign(min_of_max = test_df.apply(lambda x: 
(test_df.B[(test_df.B > x.B) & (test_df.A[test_df.A == x.A])]).min(), axis = 1))

test_df = test_df.assign(min_of_max2 = test_df.apply(lambda x : 
(test_df.min_of_max[(test_df.min_of_max > x.min_of_max) & (test_df.A[test_df.A 
== x.A])]).min(), axis = 1))

Output:

   A  B  min_of_max  min_of_max2
0  1  1         4.0          7.0
3  1  4         7.0          NaN
6  1  7         NaN          NaN
1  2  2         5.0          8.0
4  2  5         8.0          NaN
7  2  8         NaN          NaN
2  3  3         6.0          9.0
5  3  6         9.0          NaN
8  3  9         NaN          NaN

So is there any convenient way to call object from previous part of chain in second .assign ? Since using test_df.apply() in second .assign will take initial test_df without computed test_df['min_of_max']

Sorry for somewhat unreadable code in Python, I'am still figuring out how to write more clear.

Answer 1

In Pandas, run the chain of two assign calls but do so in any way that does not rely on original data frame context such as with DataFrame.apply call. Below uses a list comprehension equivalent across index values:

test_df = pd.DataFrame({'group': np.repeat([1,2,3],3), 'data': np.arange(1,10)})

(
   test_df.assign(min_of_max = lambda x: [np.min(x["data"].loc[(x["data"] > x["data"].iloc[i]) &
                                                               (x["group"] == x["group"].iloc[i])]
                                                ) for i in test_df.index.values])
          .assign(min_of_max_2 = lambda x: [np.min(x["min_of_max"].loc[(x["min_of_max"] > x["min_of_max"].iloc[i]) &
                                                                       (x["group"] == x["group"].iloc[i])]
                                                  ) for i in test_df.index.values])
)

#    group  data  min_of_max  min_of_max_2
# 0      1     1         2.0           3.0
# 1      1     2         3.0           NaN
# 2      1     3         NaN           NaN
# 3      2     4         5.0           6.0
# 4      2     5         6.0           NaN
# 5      2     6         NaN           NaN
# 6      3     7         8.0           9.0
# 7      3     8         9.0           NaN
# 8      3     9         NaN           NaN

---

However, just as you can combine the assignments in dplyr::mutate , you can do the same by combining the DataFrame.assign calls by using the lambda method (not to be confused with lambda in DataFrame.apply ).

R

test_df <- data.frame(group = rep(c(1,2,3), each = 3), data = c(1:9))

test_df %>%
  group_by(group) %>%
  mutate(min_of_max = map_dbl(data, ~data[data > .x] %>% min()),
         min_of_max_2 = map_dbl(min_of_max, ~min_of_max[min_of_max > .x] %>% min()))

# # A tibble: 9 x 4
# # Groups:   group [3]
#   group  data min_of_max min_of_max_2
#   <dbl> <int>      <dbl>        <dbl>
# 1     1     1          2            3
# 2     1     2          3          Inf
# 3     1     3        Inf          Inf
# 4     2     4          5            6
# 5     2     5          6          Inf
# 6     2     6        Inf          Inf
# 7     3     7          8            9
# 8     3     8          9          Inf
# 9     3     9        Inf          Inf

Pandas

test_df = pd.DataFrame({'group': np.repeat([1,2,3],3), 'data': np.arange(1,10)})

test_df.assign(min_of_max = lambda x: [np.min(x["data"].loc[(x["data"] > x["data"].iloc[i]) &
                                                            (x["group"] == x["group"].iloc[i])]
                                             ) for i in test_df.index.values],
               min_of_max_2 = lambda x: [np.min(x["min_of_max"].loc[(x["min_of_max"] > x["min_of_max"].iloc[i]) &
                                                                    (x["group"] == x["group"].iloc[i])]
                                               ) for i in test_df.index.values])

#    group  data  min_of_max  min_of_max_2
# 0      1     1         2.0           3.0
# 1      1     2         3.0           NaN
# 2      1     3         NaN           NaN
# 3      2     4         5.0           6.0
# 4      2     5         6.0           NaN
# 5      2     6         NaN           NaN
# 6      3     7         8.0           9.0
# 7      3     8         9.0           NaN
# 8      3     9         NaN           NaN

---

By the way, since Pandas was arguably modeled after R many years ago by Wes McKinney (see paper ), base R tends to be more translatable to Pandas. Below, within mirrors uses of assign and sapply mirrors list comprehension.

Base R

test_df <- within(test_df, {      
  min_of_max <- sapply(1:nrow(test_df), 
                       function(i) min(data[data > data[i] & 
                                            group == group[i]]))

  min_of_max_2 <- sapply(1:nrow(test_df), 
                         function(i) min(min_of_max[min_of_max > min_of_max[i] & 
                                                    group == group[i]]))      
})

test_df[c("group", "data", "min_of_max", "min_of_max_2")]

#   group data min_of_max min_of_max_2
# 1     1    1          2            3
# 2     1    2          3          Inf
# 3     1    3        Inf          Inf
# 4     2    4          5            6
# 5     2    5          6          Inf
# 6     2    6        Inf          Inf
# 7     3    7          8            9
# 8     3    8          9          Inf
# 9     3    9        Inf          Inf

Answer 2

Guess I have figured out the brief way to refer the object in previous part of chain using lambda functions. Passing into assign its argument will be treated as a data frame from previous part of chain.

(test_df.assign(min_of_max = test_df.apply(lambda x: (test_df.B[(test_df.B > x.B) &
                                                     (test_df.A[test_df.A == x.A])]).min(), axis = 1))
        .assign(min_of_max2 = lambda y: y.apply(lambda x: (y.min_of_max[(y.min_of_max > x.min_of_max) &
                                                          (y.A[y.A == x.A])]).min(), axis = 1))) 

Passing 'lambda y' in second .assign will treat y as a output from previous part in chain