如何使用data.table :: fread读取未加引号的\\ r

Question

Data I have to process has unquoted text with some additional \\r character. Files are big (500MB), copious (>600), and changing the export is not an option. Data might look like

A,B,C

blah,a,1

bloo,a\\r,b

blee,c,d

1. How can this be handled with data.table's fread ?
2. Is there a better R read CSV function for this, that's similarly performant?

Repro

library(data.table)
csv<-"A,B,C\r\n
      blah,a,1\r\n
      bloo,a\r,b\r\n
      blee,c,d\r\n"
fread(csv)

Error in fread(csv) : Expected sep (',') but new line, EOF (or other non printing character) ends field 1 when detecting types from point 0: bloo,a

Advanced repro

The simple repro might be too trivial to give a sense of scale...

samplerecs<-c("blah,a,1","bloo,a\r,b","blee,c,d")
randomcsv<-paste0(c("A,B,C",rep(samplerecs,2000000)))
write(randomcsv,file = "sample.csv")

# Naive approach
fread("sample.csv")

# Akrun's approach with needing text read first
fread(gsub("\r\n|\r", "", paste0(randomcsv,collapse="\r\n")))
#>Error in file.info(input) :  file name conversion problem -- name too long?

# Julia's approach with needing text read first
readr::read_csv(gsub("\r\n|\r", "", paste0(randomcsv,collapse="\r\n")))
#> Error: C stack usage  48029706 is too close to the limit

Answer 1

Further to @dirk-eddelbuettel & @nrussell's suggestions, a way of solving this is to is to pre-process the file. The processor could also be called within fread() but here it is performed in seperate steps:

samplerecs<-c("blah,a,1","bloo,a\r,b","blee,c,d")
randomcsv<-paste0(c("A,B,C",rep(samplerecs,2000000)))
write(randomcsv,file = "sample.csv")
# Remove errant `\r`'s with tr - shown here is the Windows R solution
shell("C:/Rtools/bin/tr.exe -d '\\r' < sample.csv > sampleNEW.csv")
fread("sampleNEW.csv")

Answer 2

We can try with gsub

fread(gsub("\r\n|\r", "", csv))
#      A B C
#1: blah a 1
#2: bloo a b
#3: blee c d

Answer 3

You can also do this with tidyverse packages, if you'd like.

> library(readr)
> library(stringr)
> read_csv(str_replace_all(csv, "\r", ""))
# A tibble: 3 × 3
      A     B     C
  <chr> <chr> <chr>
1  blah     a     1
2  bloo     a     b
3  blee     c     d

Answer 4

If you do want to do it purely in R, you could try working with connections. As long as a connection is kept open, it will start reading/writing from its previous position. Of course, this means the burden of opening and closing connections falls on you.

In the following code, the file is processed by chunks:

library(data.table)

input_csv <- "sample.csv"
in_conn <- file(input_csv)
output_csv <- "out.csv"
out_conn <- file(output_csv, "w+")
open(in_conn)

chunk_size <- 1E6
return_pattern <- "(?<=^|,|\n)([^,]*(?<!\n)\r(?!\n)[^,]*)(?=,|\n|$)"

buffer <- ""

repeat {
  new_chars <- readChar(in_conn, chunk_size)
  buffer <- paste0(buffer, new_chars)
  while (grepl("[\r\n]$", buffer, perl = TRUE)) {
    next_char <- readChar(in_conn, 1)
    buffer <- paste0(buffer, next_char)
    if (!length(next_char))
      break
  }
  chunk <- gsub("(.*)[,\n][^,\n]*$", "\\1", buffer, perl = TRUE)
  buffer <- substr(buffer, nchar(chunk) + 1, nchar(buffer))
  cleaned <- gsub(return_pattern, '"\\1"', chunk, perl = TRUE)
  writeChar(cleaned, out_conn, eos = NULL)
  if (!length(new_chars))
    break
}

writeChar('\n', out_conn, eos = NULL)

close(in_conn)
close(out_conn)

result <- fread(output_csv)

Process:

• If a chunk ends with a \\r or \\n , another character is added until it doesn't.
• Quotes are put around values containing a \\r which isn't adjacent to a \\n .
• The cleaned chunk is added to the end of another file.
• Rinse and repeat.

This code simplifies the problem by assuming no quoting is done for any field in sample.csv . It's not especially fast, but not terribly slow. Larger values for chunk_size should reduce the amount of time spent in I/O operations. If used for anything beyond this toy example, I'd strongly suggesting wrapping it in a tryCatch(...) call to make sure the files are closed afterwards.