使用read.table.ffdf将很大的csv文件读入R时，如何指定colClasses？

Question

I am trying to read a very big .csv file, of size around 20G, using the function read.table.ffdf() in the "ff" package, but had trouble in specifying the colClasses option in read.csv().

I have to specify the colClasses option because some columns in the file are labels as very long integers, eg with 11 digits. For example, two rows in the file are

86246,205,17,1719,104116343,8435,2013-03-13,12,OZ,1,2.59
86246,205,17,1719,10800749282,8435,2013-03-13,12,OZ,1,2.59 

The integer 10800749282 is too large for the type "integer" and can only be handled as either "numeric" or "character". But the value 104116343 in the above row is not large enough, so R by default will treat this column being "integer".

I tried the following but got an error. Does anyone know how to solve this problem? Highly appreciated!

dat <- read.table.ffdf(file="file.csv", FUN = "read.csv", na.strings = "", colClasses="character")

Error in ff(initdata = initdata, length = length, levels = levels, ordered = ordered, : vmode 'character' not implemented

Answer 1

As your error suggests, there is no 'character' data type implemented within the ff environment. All characters should be treated as factors . Assuming your file contains x number of columns, the below works:

dat <- read.csv.ffdf(NULL, file="file.csv", na.strings = "", colClasses=rep("factor", x))

However, what you probably need is not to import all data as factors, as it is extremely inefficient. Just import all your numerical data as 'numeric'. Assuming your first 5 columns are numeric and the rest 3 are characters:

dat <- read.csv.ffdf(NULL, file="file.csv", na.strings = "", colClasses=c(rep("numeric", 5), rep("factor", 3)))