如何从 csv 中的多个文件计算平均值

Question

Using this option in python it is possible to calculate the mean from multiple csv file

If file1.csv through file100.csv are all in the same directory, you can use this Python script:

#!/usr/bin/env python3

N = 100
mean_sum = 0
std_sum = 0
for i in range(1, N + 1):
    with open(f"file{i}.csv") as f:
        mean_sum += float(f.readline().split(",")[1])
        std_sum += float(f.readline().split(",")[1])

print(f"Mean of means: {mean_sum / N}")
print(f"Mean of stds: {std_sum / N}")

How is it possible to make it in R?

Answer 1

"all can be coded", Erik:)

It is difficult to help, if you do not provide a minimal reproducible example and describe what you attempted so far and where things go wrong for you.

The following is based on {tidyverse} ; a set of packages that work well together. I write in almost pseudo-code that should get you going. Obviously, you will have to adapt, rename to fit your project/variable names, etc.

Good luck:

library(readr)     # package to read tabular data
library(dplyr)     # main working horse to crunch data
library(purrr)     # functional programming for iterations/loops

pth <- "my-data-folder"    # provide path to your data

# create a list of file names in your folder
## you may need to fine-tune the regular pattern to select the files you look for
## full.names gives you the path/name of your data files
## \\.csv is the way to "escape" the dot of the csv type ending

fns <- list.files(path = pth, pattern = "*file.*\\.csv", full.names = TRUE)

# write a function that reads the file and calculates your stats
## you can "summarise" stats over a table

my_function <- function(.fn){
  df <- read_csv(.fn)     # read the file
  df <- df %>% 
    summarise(MEAN = mean(my-target-variable)    # calc mean of your file/data
              , SD = sd(my-target-variable)      # calc sd of the data
}

# iterate with purrr::map := take list of filenames and apply your function to each list entry
## map_dfr() provides a data frame, you can use "only" map() to get a list
## for testing purposes you can truncate the list of filenames with fns[1:3] for the
## first 3 files, other

ds <- fns %>% 
   purrr::map_dfr(.f = my_function)

ds

ds is a table with columns MEAN and SD.

Answer 2

It was kind of fun to think about making this example reproducible, so here's some code to create 100 CSVs each with five columns of random data, read them back in, and do the calculation you want. As @Ray's answer suggests, using map() and its friends is a good way to tidily iterate.

library(readr)
library(dplyr)
library(tidyr)
library(purrr)

## Make a "tmpdat" folder in the working dir if one doesn't exist
ifelse(!dir.exists(file.path("tmpdat")), dir.create(file.path("tmpdat")), FALSE)

#> [1] TRUE

## Make 100 CSV files, each with 5 columns
## of random data.
set.seed(16)

nvars <- 5

paste0("csv_", 1:100) %>%
  set_names() %>%
  map(~ replicate(n = nvars, rnorm(100, 0, 1))) %>%
  map_dfr(as_tibble, .id = "id", .name_repair = ~ paste0("v", 1:nvars)) %>%
  group_by(id) %>%
  nest() %>%
  pwalk(~ write_csv(x = .y, file = paste0("tmpdat/", .x, ".csv")))

## Get their names
filenames <- dir(path = "tmpdat",
                 pattern = "*.csv",
                 full.names = TRUE)

## Read them in and then
## 1. Calculate the mean and sd of each column in each CSV
## 2. Get the overall mean of means and mean of sds for
filenames %>%
  map_dfr(read_csv, .id = "id", col_types = cols()) %>%
  group_by(id) %>%
  summarize(across(everything(),
                   list(mean = mean, sd = sd))) %>%
  pivot_longer(-id,
               names_to = c("col", ".value"), names_sep="_") %>%
  group_by(col) %>%
  summarize(avg_mean = mean(mean),
            avg_sd = mean(sd))


#> # A tibble: 5 x 3
#>   col   avg_mean avg_sd
#>   <chr>    <dbl>  <dbl>
#> 1 v1    -0.00433  1.01 
#> 2 v2     0.00124  0.989
#> 3 v3    -0.00185  0.997
#> 4 v4     0.00431  0.991
#> 5 v5    -0.00502  0.996

If you just want a single overall mean and overall sd (rather than one for each column across all the CSVs) then this would be simpler, as you could just pivot the CSV variables into a single vector grouped by file id and take the mean and sd of that.