Calculating upper and lower confidence intervals by group in dplyr summarise()

Question

I am trying to make a table that shows N (number of observations), percent frequency (of answers > 0), and the lower and upper confidence intervals for percent frequency, and I want to group this by type.

Example of data

dat <- data.frame(
  "type" = c("B","B","A","B","A","A","B","A","A","B","A","A","A","B","B","B"),
  "num" = c(3,0,0,9,6,0,4,1,1,5,6,1,3,0,0,0)
)

Expected output (with values filled in):

Type   N   Percent   Lower 95% CI   Upper 95% CI
A
B

Attempt

library(dplyr)
library(qwraps2)

table<-dat %>%
  group_by(type) %>%
  summarise(N=n(),
            mean.ci = mean_ci(dat$num),
            "Percent"=n_perc(num > 0))

This worked to get N and percent frequency, but returned an error: "Column must be length 1 (a summary value), not 3" when I added in mean_ci

The second code I tried, found here :

table2<-dat %>%
  group_by(type) %>%
  summarise(N.num=n(),
            mean.num = mean(dat$num),
            sd.num = sd(dat$num),
            "Percent"=n_perc(num > 0)) %>%
  mutate(se.num = sd.num / sqrt(N.num),
         lower.ci = 100*(mean.num - qt(1 - (0.05 / 2), N.num - 1) * se.num),
         upper.ci = 100*(mean.num + qt(1 - (0.05 / 2), N.num - 1) * se.num))

# A tibble: 2 x 8
#  type  N.num mean.num sd.num Percent        se.num lower.ci upper.ci
# <fct> <int>    <dbl>  <dbl> <chr>           <dbl>    <dbl>    <dbl>
#1 A         8     2.44   2.83 "6 (75.00\\%)"   1.00     7.35     480.
#2 B         8     2.44   2.83 "4 (50.00\\%)"   1.00     7.35     480.

This gave me an output, but the confidence intervals are not logical.

Answer 1

The output of mean_ci is a vector of length 3. This is maybe unexpected because the package has added a print method so that when you see this in the console it looks like a single character value and not a numeric length > 1 vector. But, you can see the underlying data structure by looking at str .

mean_ci(dat$num) %>% str
 # 'qwraps2_mean_ci' Named num [1:3] 2.44 1.05 3.82
 # - attr(*, "names")= chr [1:3] "mean" "lcl" "ucl"
 # - attr(*, "alpha")= num 0.05

In summarize, each element of each column of the output needs to be length 1, so providing a length 3 object for summarize to put in a single "cell" (column element) results in an error. A workaround is to put the length 3 vector in a list, so that it is now a length 1 list. Then you can use unnest_wider to separate it into 3 columns (and therefore making the table "wider")

library(tidyverse)

dat %>%
  group_by(type) %>%
  summarise( N=n(),
            mean.ci = list(mean_ci(num)),
            "Percent"= n_perc(num > 0)) %>% 
  unnest_wider(mean.ci)
# # A tibble: 2 x 6
#   type      N  mean   lcl   ucl Percent       
#   <fct> <int> <dbl> <dbl> <dbl> <chr>         
# 1 A         8  2.25 0.523  3.98 "6 (75.00\\%)"
# 2 B         8  2.62 0.344  4.91 "4 (50.00\\%)"

Answer 2

IceCreamToucan's answer is very good. I'm posting this answer to offer a different way to present the information.

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(qwraps2)

dat <- data.frame("type" = c("B","B","A","B","A","A","B","A","A","B","A","A","A","B","B","B"),
                  "num"  = c(3,0,0,9,6,0,4,1,1,5,6,1,3,0,0,0))

When building the dplyr::summarize call you can use the qwraps2::frmtci call to format the output of qwraps2::mean_ci into a character string of length one.

I would also recommend using the data pronoun .data so you can be explicit about the variables to summarize.

dat %>%
  dplyr::group_by(type) %>%
  dplyr::summarize(N = n(),
                   mean.ci = qwraps2::frmtci(qwraps2::mean_ci(.data$num)),
                   Percent = qwraps2::n_perc(.data$num > 0))
#> `summarise()` ungrouping output (override with `.groups` argument)
#> # A tibble: 2 x 4
#>   type      N mean.ci           Percent       
#>   <chr> <int> <chr>             <chr>         
#> 1 A         8 2.25 (0.52, 3.98) "6 (75.00\\%)"
#> 2 B         8 2.62 (0.34, 4.91) "4 (50.00\\%)"

^{Created on 2020-09-15 by the reprex package (v0.3.0)}

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Catalina 10.15.6      
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/Denver              
#>  date     2020-09-15                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.0)
#>  backports     1.1.9   2020-08-24 [1] CRAN (R 4.0.2)
#>  callr         3.4.4   2020-09-07 [1] CRAN (R 4.0.2)
#>  cli           2.0.2   2020-02-28 [1] CRAN (R 4.0.0)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.0)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 4.0.0)
#>  devtools      2.3.1   2020-07-21 [1] CRAN (R 4.0.2)
#>  digest        0.6.25  2020-02-23 [1] CRAN (R 4.0.0)
#>  dplyr       * 1.0.2   2020-08-18 [1] CRAN (R 4.0.2)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.0)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  generics      0.0.2   2018-11-29 [1] CRAN (R 4.0.0)
#>  glue          1.4.2   2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.8     2019-03-20 [1] CRAN (R 4.0.0)
#>  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.0)
#>  knitr         1.29    2020-06-23 [1] CRAN (R 4.0.0)
#>  lifecycle     0.2.0   2020-03-06 [1] CRAN (R 4.0.0)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 4.0.0)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 4.0.0)
#>  pillar        1.4.6   2020-07-10 [1] CRAN (R 4.0.2)
#>  pkgbuild      1.1.0   2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.0.0)
#>  pkgload       1.1.0   2020-05-29 [1] CRAN (R 4.0.0)
#>  prettyunits   1.1.1   2020-01-24 [1] CRAN (R 4.0.0)
#>  processx      3.4.4   2020-09-03 [1] CRAN (R 4.0.2)
#>  ps            1.3.4   2020-08-11 [1] CRAN (R 4.0.2)
#>  purrr         0.3.4   2020-04-17 [1] CRAN (R 4.0.0)
#>  qwraps2     * 0.5.0   2020-09-14 [1] local         
#>  R6            2.4.1   2019-11-12 [1] CRAN (R 4.0.0)
#>  Rcpp          1.0.5   2020-07-06 [1] CRAN (R 4.0.0)
#>  remotes       2.2.0   2020-07-21 [1] CRAN (R 4.0.2)
#>  rlang         0.4.7   2020-07-09 [1] CRAN (R 4.0.2)
#>  rmarkdown     2.3     2020-06-18 [1] CRAN (R 4.0.0)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 4.0.0)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  stringi       1.5.3   2020-09-09 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  testthat      2.3.2   2020-03-02 [1] CRAN (R 4.0.0)
#>  tibble        3.0.3   2020-07-10 [1] CRAN (R 4.0.2)
#>  tidyselect    1.1.0   2020-05-11 [1] CRAN (R 4.0.0)
#>  usethis       1.6.1   2020-04-29 [1] CRAN (R 4.0.0)
#>  utf8          1.1.4   2018-05-24 [1] CRAN (R 4.0.0)
#>  vctrs         0.3.4   2020-08-29 [1] CRAN (R 4.0.2)
#>  withr         2.2.0   2020-04-20 [1] CRAN (R 4.0.0)
#>  xfun          0.17    2020-09-09 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library