R data.table: using fread on all .csv files in folder skipping the last line of each

Question

I have hundreds of .csv files I need to read in using fread and save as one data table. The basic structure is the same for each .csv. There is header info that needs to be skipped (easy using skip = ). I am having difficulty with skipping the last line of each .csv file. Each .csv file has a different number of rows.

If I have only one file in the Test folder, this script perfectly skips the first rows (using skip = ) and the last row (using nrows = ):

file <- list.files("Q:/Test/", full.names=TRUE)
all <- fread(file, skip = 7, select = c(1:7,9),
             nrows = length(readLines(file))-9)

When saving multiple files in the Test folder, this is the code I tried:

file <- list.files("Q:/Test/", full.names=TRUE)
L <- lapply(file, fread, skip = 7, select = c(1:7,9),
        nrows = length(readLines(file))-9)
dt <- rbindlist(L)

It doesn't create L and gives me this error:

Error in file(con, "r") : invalid 'description' argument

Any ideas on how to skip the last row of each .csv when each .csv has a different number of rows?

I am using data.table version 1.9.6. Thanks.

Answer 1

It's a bit late, but here's what worked for me:

library(data.table)

fnames <- dir("path", pattern = "csv")

read_data <- function(z){
  dat <- fread(z, skip = 1, select = 1)
  return(dat[1:(nrow(dat)-1),])
}

datalist <- lapply(fnames, read_data)

bigdata <- rbindlist(datalist, use.names = TRUE)

Here path refers to the directory that you're looking into. I'm assuming that the names are similar for all read files, if not, you can always define a new name for bigdata using names . Hope this helps!