使用 dtplyr 的 group-filter-select 翻译不正确

Question

A group-filter-select is easy to perform with dplyr. In the example below, we have some data on companies for different quarters this year. I now want to filter to the first quarter of companies which don't have data for the fourth quarter (in this case, the second company), dropping the quarter-label.

df <- data.frame(companyId = c(rep(1, 4),
                               rep(2, 3),
                               rep(3, 4)),
                 Quarter = c(1:4, 1:3, 1:4),
                 Year = 2019)

q <- 4                 

df %>%
  group_by(
    companyId,
  ) %>%
  filter(
    Quarter == 1 &
      !(q %in% Quarter)
  ) %>%
  select(companyId,
         Year)

> # A tibble: 1 x 3
> # Groups:   companyId, Ticker [1]
>   companyId  Year
>       <dbl> <dbl>
> 1         2  2019

However, doing the same with dtplyr returns an empty table:

dt <- lazy_dt(data.table(companyId = c(rep(1, 4),
                                       rep(2, 3),
                                       rep(3, 4)),
                         Quarter = c(1:4, 1:3, 1:4),
                         Year = 2019))

q <- 4

dt %>%
  group_by(
    companyId
  ) %>%
  filter(
    Quarter == 1 &
      !(q %in% Quarter)
  ) %>%
  select(companyId
         Year)

> Source: local data table [?? x 3]
> Call:   `_DT1`[Quarter == 1 & !(q %in% Quarter), .(companyId, 
>     Year)]
> 
> # ... with 3 variables: companyId <dbl>, Year <dbl>
> 
> # Use as.data.table()/as.data.frame()/as_tibble() to access results

What's odd is the displayed translation:

`_DT1`[Quarter == 1 & !(q %in% Quarter),
       .(companyId, Year)]

which is incorrect. As described in the dtplyr'sown docs , the correct call would need to use a filtered .SD :

`_DT1`[, .SD[Quarter == 1 & !(q %in% Quarter)],
       by = .(companyId),
       .SDcols = c("Year")]

(the by-columns are automatically included, so .SDcols should omit them to avoid duplication)

Interestingly, if we omit the select , the translation (and therefore output) is correct:

dt %>%
  group_by(
    companyId
  ) %>%
  filter(
    Quarter == 1 &
      !(q %in% Quarter)
  )

> Source: local data table [?? x 4]
> Call:   `_DT2`[, .SD[Quarter == 1 & !(q %in% Quarter)], 
>     keyby = .(companyId)]
> 
>   companyId Quarter  Year
>       <dbl>   <int> <dbl>
> 1         2       1  2019

Therefore, as a workaround, I can perform an as.data.table() prior to the select . This works, but throws an annoying warning:

dt %>%
  group_by(
    companyId
  ) %>%
  filter(
    calendarQuarter == 1 &
      !(q %in% calendarQuarter)
  ) %>%
  as.data.table() %>%
  select(companyId,
         calendarYear)

>    companyId calendarYear
> 1:         2         2019
> Warning message:
> You are using a dplyr method on a raw data.table, which will call the data frame implementation,
> and is likely to be inefficient.
> * 
> * To suppress this message, either generate a data.table translation with `lazy_dt()` or convert
> * to a data frame or tibble with `as.data.frame()`/`as_tibble()`.

I have a hard time thinking this is expected behavior, but would like to check here before throwing this on the dtplyr Github tracker.

Answer 1

This is currently a bug in dtplyr . I have posted it to the package's Github .