在 for 循环中为文件列表添加一个包含文件名的新列

Question

I have a time series data. I stored the data in txt files under daily subfolders in Monthly folders.

setwd(".../2018/Jan")
parent.folder <-".../2018/Jan"  
sub.folders <- list.dirs(parent.folder, recursive=TRUE)[-1] #To read the sub-folders under parent folder
r.scripts <- file.path(sub.folders)
A_2018 <- list()
for (j in seq_along(r.scripts)) {
  A_2018[[j]] <- dir(r.scripts[j],"\\.txt$")}

Of these.txt files, I removed some of the files which I don't want to use for the further analysis, using the following code.

trim_to_two <- function(x) {
  runs = rle(gsub("^L1_\\d{4}_\\d{4}_","",x))
  return(cumsum(runs$lengths)[which(runs$lengths > 2)] * -1)
}

A_2018_new <- list()
for (j in seq_along(A_2018)) {
  A_2018_new[[j]] <- A_2018[[j]][trim_to_two(A_2018[[j]])]
  }

Then, I want to make a rowbind by for loop for the whole.txt files. Before that, I would like to remove some lines in each txt file, and add one new column with file name. The following is my code.

for (i in 1:length(A_2018_new)) {
  
  for (j in 1:length(A_2018_new[[i]])){
       
    filename <- paste(str_sub(A_2018_new[[i]][j], 1, 14))
        
    assign(filename, read_tsv(complete_file_name, skip = 14, col_names = FALSE), 
           )
    
    Y <- r.scripts %>% str_sub(46, 49)
    MD <- r.scripts %>% str_sub(58, 61)
    HM <- filename %>% str_sub(9, 12)
    Turn <- filename %>% str_sub(14, 14)
    time_minute <- paste(Y, MD, HM, sep="-")
    
    Map(cbind, filename, SampleID = names(filename))
    }
} 

But I didn't get my desired output. I tried to code using other examples. Could anyone help to explain what my code is missing.

Answer 1

Your code seems overly complex for what it is doing. Your problem is however not 100% clear (eg what is the pattern in your file names that determine what to import and what not?). Here are some pointers that would greatly simplify the code, and likely avoid the issue you are having.

Use lapply() or map() from the purrr package to iterate instead of a for loop. The benefit is that it places the different data frames in a list and you don't need to assign multiple data frames into their own objects in the environment. Since you tagged the tidyverse, we'll use the purrr functions.

library(tidyverse)

You could for instance retrieve the txt file paths, using something like

txt_files <- list.files(path = 'data/folder/', pattern = "txt$", full.names = TRUE) # Need to remove those files you don't with whatever logic applies

and then use map() with read_tsv() from readr like so:

mydata <- map(txt_files, read_tsv)

Then for your manipulation, you can again use lapply() or map() to apply that manipulation to each data frame. The easiest way is to create a custom function, and then apply it to each data frame:

my_func <- function(df, filename) {
  df |>
    filter(...) |> # Whatever logic applies here
    mutate(filename = filename)
}

and then use map2() to apply this function, iterating through the data and filenames, and then list_rbind() to bind the data frames across the rows.

mydata_output <- map2(mydata, txt_files, my_func) |>
  list_rbind()