循环多个 excel 文件以创建不同的数据帧，执行分组并保存为 R 中的单个 df

Question

I am new in R and I have a doubt, in case you can help.

I have several excel files in one folder. They belong to different filials but have the same structure.
I would like to loop over them, load into R as a dataframe, perform a group by and save everything in a single dataframe and export as a single file. Would this be possible?

By looking at several answers here I did this:

# Load the data as different dataframes

library(tidyverse)
library(readxl)

f <- list.files(pattern="xlsx")

myfiles = lapply(f, read_excel)


for (i in 1:length(f)) assign(f[i], read_excel(f[i], sheet = "Deutsch", skip=7), data.frame(f[i]))

I have them saved as single dataframes, I don't know how to access them all together, so I manually created a list:

list_df = list(filialAA.xlsx, filialAB.xlsx,filianAC.xlsx,filianAD.xlsx,filianAE.xlsx...etc)

Then I created a group by to perform some calculations:

for (i in 1:length(list_df))
{
  list_df[i] %>% 
    group_by(ABC) %>% 
    summarise(`Revenue in EUR` = sum(`Revenue in EUR`),
              `Weight in KG` = sum(`Weight in KG`),
              `Number of Materials` = length(`Materials`),
              `Avg of deliveries` = mean(`Deliveries`))
}

If I do this for each dataframe, it works. But inside this loop it does not. Could you help me to loop over all dataframes, perform this group by and gather together in one single file? Is it possible?

Thanks a lot for your attention!

EDIT: To include a dummy data sample:

> dput(df1)

structure(list(Materials = c("11575358", "75378378", "21333333", 
"02469984", "05465478", "05645648"), Deliveries = c(8, 1, 12, 
5, 1, 1), ABC = c("C", "A", "C", "B", "C", "C"), `Revenue in EUR` = c(6179, 
1804802.46, 3768.04, 9e+05, 1597.5, 1544.55), `Weight in KG` = c(16.6, 
4.695625, 19, 9.14625, 2.74041666666667, 1.44208333333333)), row.names = c(NA, 
-6L), class = c("tbl_df", "tbl", "data.frame"))

> dput(df2)

structure(list(Materials = c("48654798", "05465489", "04598496", 
"08789453", "01589494", "06459849", "54694985", "65498848"), 
    Deliveries = c(24, 6, 32, 3, 11, 30, 45, 2), ABC = c("C", 
    "B", "C", "B", "C", "A", "A", "C"), `Revenue in EUR` = c(5509, 
    506978, 3978.04, 7e+05, 1597.5, 1200258, 2406975, 4059), 
    `Weight in KG` = c(29.6, 19, 24, 9.14625, 2.74041666666667, 
    50, 60, 10)), row.names = c(NA, -8L), class = c("tbl_df", 
"tbl", "data.frame"))

Original excel is xlsx format, have from 5000 to 15000 rows, about 20 features, 7 tabs. There are 22 excel files to loop over.

Answer 1

Ok it could have some error due I have not your files, but try something like this:

# first of, write down your files in xlsx. I use xlsx because I prefere it
#but you should already have them
xlsx::write.xlsx2(df1,"df1.xlsx")
xlsx::write.xlsx2(df1,"df2.xlsx")

library(tidyverse)
library(readxl)

# here you get all the xlsx files
f <- list.files(pattern="xlsx")  
f
[1] "df1.xlsx" "df2.xlsx"

# an empty list
listed <- list()
# loop that populate the empty list with your files
for (i in f) { 
  listed[[i]] <- read_excel(i, sheet = "Sheet1" # , skip = 7  
                            )
  print(paste0("read the", i," file")) # here it says what it's doing
}

 listed
$df1.xlsx
# A tibble: 6 x 6
  ...1  Materials Deliveries ABC   `Revenue in EUR` `Weight in KG`
  <chr> <chr>          <dbl> <chr>            <dbl>          <dbl>
1 1     11575358           8 C                6179           16.6 
2 2     75378378           1 A             1804802.           4.70
3 3     21333333          12 C                3768.          19   
4 4     02469984           5 B              900000            9.15
5 5     05465478           1 C                1598.           2.74
6 6     05645648           1 C                1545.           1.44

$df2.xlsx
# A tibble: 6 x 6
  ...1  Materials Deliveries ABC   `Revenue in EUR` `Weight in KG`
  <chr> <chr>          <dbl> <chr>            <dbl>          <dbl>
1 1     11575358           8 C                6179           16.6 
2 2     75378378           1 A             1804802.           4.70
3 3     21333333          12 C                3768.          19   
4 4     02469984           5 B              900000            9.15
5 5     05465478           1 C                1598.           2.74
6 6     05645648           1 C                1545.           1.44

# now lapply to each element of the list, the summary, creating a new list
list_result <- lapply(listed, function(x) x %>% 
                                          group_by(ABC) %>% 
                                          summarise(
                          `Revenue in EUR` = sum(`Revenue in EUR`),
                          `Weight in KG` = sum(`Weight in KG`),
                          `Number of Materials` = length(`Materials`),
                          `Avg of deliveries` = mean(`Deliveries`)))

# put the result in a data.frame  
do.call(rbind,list_result)
# A tibble: 6 x 5
  ABC   `Revenue in EUR` `Weight in KG` `Number of Materials` `Avg of deliveries`
* <chr>            <dbl>          <dbl>                 <int>               <dbl>
1 A             1804802.           4.70                     1                 1  
2 B              900000            9.15                     1                 5  
3 C               13089.          39.8                      4                 5.5
4 A             1804802.           4.70                     1                 1  
5 B              900000            9.15                     1                 5  
6 C               13089.          39.8                      4                 5.5

Answer 2

You may also use purrr::map suitably

map_dfr(list_df, ~(. %>% 
    group_by(ABC) %>% 
    summarise(`Revenue in EUR` = sum(`Revenue in EUR`),
              `Weight in KG` = sum(`Weight in KG`),
              `Number of Materials` = length(`Materials`),
              `Avg of deliveries` = mean(`Deliveries`))))

It will rbind the results simultaneously.

Even after storing files in myfiles you can use the following syntax

library(janitor)
map_dfr(myfiles, ~(.[-c(1:5),] %>% row_to_names(1) %>% 
                     group_by(ABC) %>% 
                     summarise(`Revenue in EUR` = sum(as.numeric(`Revenue in EUR`)),
                               `Weight in KG` = sum(as.numeric(`Weight in KG`)),
                               `Number of Materials` = length(`Materials`),
                               `Avg of deliveries` = mean(as.numeric(`Deliveries`)))
                   %>% ungroup()))

results with your given files

# A tibble: 6 x 5
  ABC   `Revenue in EUR` `Weight in KG` `Number of Materials` `Avg of deliveries`
  <chr>            <dbl>          <dbl>                 <int>               <dbl>
1 A             1804802.           4.70                     1                 1  
2 B              900000            9.15                     1                 5  
3 C               13089.          39.8                      4                 5.5
4 A             3607233          110                        2                37.5
5 B             1206978           28.1                      2                 4.5
6 C               15144.          66.3                      4                17.2

Answer 3

I like to write functions, so I would do it like this (although longer it creates a more stable environment to modify/debug when required).

# Main Function
main_function <- function(import, name){
 main_function.create_path() -> path
 main_function.create_output() -> output
 for(file in list.files(path){
  if(!str_detect(file, 'csv')){
   next
  }
  read_excel(file, sheet = "Deutsch", skip = 7) -> data
  main_function.calculate_values(data) -> data.values
  main_function.append_values(file, data, data.values, output) -> output
 }
 main_function.export(path, output, name)
 if(import){
  assign('values', output, envir = .Globalenv)
 }
}
    
# Functions
main_function.export <- function(path, output, name){
 write.csv(output, file = paste0(path, name, '.csv'))
 }
 
main_function.append_values <- function(file, data, data.values,   output){
 # This will create a row in the output file with the name of the file
 # without the .csv at the end in the first column and put in the 
 # calculated data in the other columns
 str_extract(file, ".+(?=.csv)") -> output[nrow(output) + 1, 'file'] 
 for(col in colnames(data.values)){
   data.values[,col] -> output[nrow(output), col]
 return(output)
 }

main_function.calculate_values <- function(data){
 data %>% group_by(ABC) %>%
  summarize(`Revenue in EUR` = sum(`Revenue in Eur`, na.rm=TRUE),
            ....) -> data
 return(data)
 }
 
main_function.create_path <- function(){
 '<path to files>' -> path
 return(path) 
 }
   
main_function.create_output <- function(){
 data.frame('file' = as.character(NA), 'Revenue in EUR' = 0, 
  'Weight in KG' = 0, 'Number of Materials' = 0, 'Avg of deliveries' = 0) -> output
 return(output)
 }

This will create the main_function that when called will cycle through all of the files listed in the path given and read it, process it, save it to output which which will be saved in the same path with the name you give it. If you set import to TRUE it will also save the output