[英]Read all files in a folder and apply a function to each data frame
I am doing a relatively simple piece of analysis which I have put into a function, on all the files in a particular folder.我正在对特定文件夹中的所有文件进行一个相对简单的分析,并将其放入一个函数中。 I was wondering whether anyone had any tips to help me automate the process on a number of different folders.我想知道是否有人有任何提示可以帮助我在许多不同的文件夹上自动化该过程。
files <- (Sys.glob("*.csv"))
...which I found from Using R to list all files with a specified extension ...我从使用 R 中找到的列出具有指定扩展名的所有文件
And then the following code reads all those files into R.然后以下代码将所有这些文件读入 R。
listOfFiles <- lapply(files, function(x) read.table(x, header = FALSE))
…from Manipulating multiple files in R ...来自在 R 中操作多个文件
But the files seem to be read in as one continuous list and not individual files… how can I change the script to open all the csv files in a particular folder as individual dataframes?但是这些文件似乎是作为一个连续列表而不是单个文件读入的……我如何更改脚本以将特定文件夹中的所有 csv 文件作为单个数据帧打开?
Secondly, assuming that I can read all the files in separately, how do I complete a function on all these dataframes in one go.其次,假设我可以单独读取所有文件,我如何一次性完成所有这些数据帧的功能。 For example, I have created four small dataframes so I can illustrate what I want:例如,我创建了四个小数据框,所以我可以说明我想要什么:
Df.1 <- data.frame(A = c(5,4,7,6,8,4),B = (c(1,5,2,4,9,1))) Df.2 <- data.frame(A = c(1:6),B = (c(2,3,4,5,1,1))) Df.3 <- data.frame(A = c(4,6,8,0,1,11),B = (c(7,6,5,9,1,15))) Df.4 <- data.frame(A = c(4,2,6,8,1,0),B = (c(3,1,9,11,2,16)))
I have also made up an example function:我还编写了一个示例函数:
Summary<-function(dfile){
SumA<-sum(dfile$A)
MinA<-min(dfile$A)
MeanA<-mean(dfile$A)
MedianA<-median(dfile$A)
MaxA<-max(dfile$A)
sumB<-sum(dfile$B)
MinB<-min(dfile$B)
MeanB<-mean(dfile$B)
MedianB<-median(dfile$B)
MaxB<-max(dfile$B)
Sum<-c(sumA,sumB)
Min<-c(MinA,MinB)
Mean<-c(MeanA,MeanB)
Median<-c(MedianA,MedianB)
Max<-c(MaxA,MaxB)
rm(sumA,sumB,MinA,MinB,MeanA,MeanB,MedianA,MedianB,MaxA,MaxB)
Label<-c("A","B")
dfile_summary<-data.frame(Label,Sum,Min,Mean,Median,Max)
return(dfile_summary)}
I would ordinarily use the following command to apply the function to each individual dataframe.我通常会使用以下命令将该函数应用于每个单独的数据帧。
Df1.summary<-Summary(dfile) df1.summary<-Summary(dfile)
Is there a way instead of applying the function to all the dataframes, and use the titles of the dataframes in the summary tables (ie Df1.summary).有没有办法而不是将函数应用于所有数据框,并在汇总表(即 Df1.summary)中使用数据框的标题。
Many thanks,非常感谢,
Katie凯蒂
On the contrary, I do think working with list
makes it easy to automate such things.相反,我确实认为使用list
可以很容易地自动化这些事情。
Here is one solution (I stored your four dataframes in folder temp/
).这是一种解决方案(我将您的四个数据帧存储在文件夹temp/
)。
filenames <- list.files("temp", pattern="*.csv", full.names=TRUE)
ldf <- lapply(filenames, read.csv)
res <- lapply(ldf, summary)
names(res) <- substr(filenames, 6, 30)
It is important to store the full path for your files (as I did with full.names
), otherwise you have to paste the working directory, eg存储文件的完整路径很重要(就像我对full.names
所做的full.names
),否则你必须粘贴工作目录,例如
filenames <- list.files("temp", pattern="*.csv")
paste("temp", filenames, sep="/")
will work too.也会工作。 Note that I used substr
to extract file names while discarding full path.请注意,我使用substr
来提取文件名,同时丢弃完整路径。
You can access your summary tables as follows:您可以按如下方式访问汇总表:
> res$`df4.csv`
A B
Min. :0.00 Min. : 1.00
1st Qu.:1.25 1st Qu.: 2.25
Median :3.00 Median : 6.00
Mean :3.50 Mean : 7.00
3rd Qu.:5.50 3rd Qu.:10.50
Max. :8.00 Max. :16.00
If you really want to get individual summary tables, you can extract them afterwards.如果你真的想得到单独的汇总表,你可以在之后提取它们。 Eg,例如,
for (i in 1:length(res))
assign(paste(paste("df", i, sep=""), "summary", sep="."), res[[i]])
usually i don't use for loop in R, but here is my solution using for loops and two packages : plyr and dostats通常我不在 R 中使用 for 循环,但这是我使用 for 循环和两个包的解决方案: plyr和dostats
plyr is on cran and you can download dostats on https://github.com/halpo/dostats (may be using install_github from Hadley devtools package) plyr在 cran 上,您可以在https://github.com/halpo/dostats上下载dostats (可能正在使用 Hadley devtools包中的 install_github)
Assuming that i have your first two data.frame (Df.1 and Df.2) in csv files, you can do something like this.假设我在 csv 文件中有你的前两个 data.frame(Df.1 和 Df.2),你可以做这样的事情。
require(plyr)
require(dostats)
files <- list.files(pattern = ".csv")
for (i in seq_along(files)) {
assign(paste("Df", i, sep = "."), read.csv(files[i]))
assign(paste(paste("Df", i, sep = ""), "summary", sep = "."),
ldply(get(paste("Df", i, sep = ".")), dostats, sum, min, mean, median, max))
}
Here is the output这是输出
R> Df1.summary
.id sum min mean median max
1 A 34 4 5.6667 5.5 8
2 B 22 1 3.6667 3.0 9
R> Df2.summary
.id sum min mean median max
1 A 21 1 3.5000 3.5 6
2 B 16 1 2.6667 2.5 5
Here is a tidyverse
option that might not the most elegant, but offers some flexibility in terms of what is included in the summary:这是一个tidyverse
选项,它可能不是最优雅的,但在摘要中包含的内容方面提供了一些灵活性:
library(tidyverse)
dir_path <- '~/path/to/data/directory/'
file_pattern <- 'Df\\.[0-9]\\.csv' # regex pattern to match the file name format
read_dir <- function(dir_path, file_name){
read_csv(paste0(dir_path, file_name)) %>%
mutate(file_name = file_name) %>% # add the file name as a column
gather(variable, value, A:B) %>% # convert the data from wide to long
group_by(file_name, variable) %>%
summarize(sum = sum(value, na.rm = TRUE),
min = min(value, na.rm = TRUE),
mean = mean(value, na.rm = TRUE),
median = median(value, na.rm = TRUE),
max = max(value, na.rm = TRUE))
}
df_summary <-
list.files(dir_path, pattern = file_pattern) %>%
map_df(~ read_dir(dir_path, .))
df_summary
# A tibble: 8 x 7
# Groups: file_name [?]
file_name variable sum min mean median max
<chr> <chr> <int> <dbl> <dbl> <dbl> <dbl>
1 Df.1.csv A 34 4 5.67 5.5 8
2 Df.1.csv B 22 1 3.67 3 9
3 Df.2.csv A 21 1 3.5 3.5 6
4 Df.2.csv B 16 1 2.67 2.5 5
5 Df.3.csv A 30 0 5 5 11
6 Df.3.csv B 43 1 7.17 6.5 15
7 Df.4.csv A 21 0 3.5 3 8
8 Df.4.csv B 42 1 7 6 16
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.