read.csv2.ffdf正在导入一个数字（浮点）变量作为因子

Question

I have a couple weeks working with the ff package and it has been working great so far, but today I realized that a variable that should be numeric is being readed as a factor. The data has about 900k rows and 800 col, so it's not easy to control that every column gets the class that it should...

matff <- read.csv2.ffdf(file = name,encoding = "UTF-8",next.rows=150000,colClasses=NA)

I would like to know why may this be happening and an idea on how to fix it.

Thanks.

Answer 1

Your data has some columns which are clearly texts and not numeric data as you expect it.

You can use the transFUN argument to read.csv2.ffdf to solve your decimal problem. As in

transFUN=function(x){
  x$mycolumn <- as.numeric(gsub(",", ".", as.character(x$mycolumn)))
  x
}

Or use the appropriate read.table arguments.

Answer 2

Now it should work:

# matff <- data.frame(Col=c('a','b','c'),Mix1=c('a','1.2','c'),Mix2=c(1.1,2.1,3),Num1=c('1.2','2.3','3.4'),Num2=c('1,2','2,3','3,4')) # Data example

func <- function(x) {
 if (class(x) != 'numeric') {
  x <- levels(x)[x]
  if (length(grep('[a-zA-Z]',x,invert=T)) == length(x)) { x <- as.real(gsub(',','\\.',x)) }
  else { x <- factor(x) }
 }
 x
}

for (i in 1:ncol(matff)) {
 matff[,i] <- func(matff[,i])
}