Modify, extract, and concatenate list sub-elements into a data.frame in R with tidyverse

Question

I'm trying to find an elegant way to work with list structures in R. In particular, in this case, I'd like to extract sub-elements from a list, modify them based on their associated data in that list, and concatenate them into a data frame. Perhaps easier with an example:

mystruct <- structure(list(dataset1 = structure(list(data1 = structure(list(
    a = c(1, 2, 3), b = c(4, 5, 6)), .Names = c("a", "b"), row.names = c(NA, 
-3L), class = "data.frame"), data2 = c("a", "b", "c", "d", "e"
)), .Names = c("data1", "data2")), dataset2 = structure(list(
    data1 = structure(list(a = c(7, 8, 9), b = c(10, 11, 12)), .Names = c("a", 
    "b"), row.names = c(NA, -3L), class = "data.frame"), data2 = c("f", 
    "g", "h", "i", "j")), .Names = c("data1", "data2"))), .Names = c("dataset1", 
"dataset2"))

I can concatenate data1 elements like this:

> mystruct %>% map_dfr(~.x$data1)
  a  b
1 1  4
2 2  5
3 3  6
4 7 10
5 8 11
6 9 12

But I would like to add a "dataset" column, which is populated by the name of the list element from whence the data was taken:

  dataset    a  b
1 dataset1   1  4
2 dataset1   2  5
3 dataset1   3  6
4 dataset2   7 10
5 dataset2   8 11
6 dataset2   9 12

Is there a way to do this nicely with the tidyverse? I'd also be open to data.table solutions.

Thanks, Allie

Answer 1

Provide an .id parameter to map_df , which will create a column giving the name of the list:

map_df(mystruct, 'data1', .id='dataset')

#   dataset a  b
#1 dataset1 1  4
#2 dataset1 2  5
#3 dataset1 3  6
#4 dataset2 7 10
#5 dataset2 8 11
#6 dataset2 9 12

Or map_dfr should work as well:

map_dfr(mystruct, 'data1', .id='dataset')

Answer 2

map_dfr has an .id argument:

mystruct %>% map_dfr(~ .x$data1, .id = "id")

giving:

        id a  b
1 dataset1 1  4
2 dataset1 2  5
3 dataset1 3  6
4 dataset2 7 10
5 dataset2 8 11
6 dataset2 9 12

Answer 3

Restructure as a "tidy" table with list columns...

library(data.table)
tabstruct = rbindlist(lapply(mystruct, lapply, list), id = TRUE)
#         .id        data1     data2
# 1: dataset1 <data.frame> a,b,c,d,e
# 2: dataset2 <data.frame> f,g,h,i,j

Then "unnest" data1:

tabstruct[, rbindlist(setNames(data1, .id), id=TRUE)]

#         .id a  b
# 1: dataset1 1  4
# 2: dataset1 2  5
# 3: dataset1 3  6
# 4: dataset2 7 10
# 5: dataset2 8 11
# 6: dataset2 9 12

Or unnest data2:

tabstruct[, .(val = unlist(data2)), by=.id]
#          .id val
#  1: dataset1   a
#  2: dataset1   b
#  3: dataset1   c
#  4: dataset1   d
#  5: dataset1   e
#  6: dataset2   f
#  7: dataset2   g
#  8: dataset2   h
#  9: dataset2   i
# 10: dataset2   j

Answer 4

Here is an option to do this on multiple datasets in the list

map(c('data1', 'data2'), ~  
         map2_df(mystruct, .x, ~ .x[[.y]], .id = 'id'))
#[[1]]
#        id a  b
#1 dataset1 1  4
#2 dataset1 2  5
#3 dataset1 3  6
#4 dataset2 7 10
#5 dataset2 8 11
#6 dataset2 9 12

#[[2]]
# A tibble: 5 x 3
#  id    dataset1 dataset2
#  <chr> <chr>    <chr>   
#1 1     a        f       
#2 1     b        g       
#3 1     c        h       
#4 1     d        i       
#5 1     e        j