How to delete ffdf objects directories in R?

Question

I am using the ffdf package to do some data pre-processing. My work computer has 4 CPU cores and 8 Gb of RAM, and I can handle about 0.2-0.3 billion data points, which is really wonderful.

However, I have another constraint. The large ffdf objects use up my computer's disk space. When I combine 5 tables into one, and add 20 more columns on it, I got a message "no disk space" .

So, I want to remove un-used ffdf objects on my disk drive. Can I do it without exiting the R session? (I have read, write and execute access on the folder.)

I have tried everything I know, like unlink() , file.remove() , delete() , close() , finalize() , finalizer() , ffdrop() ".

delete_dir <- "d:/ff/t1_pre"
deletephrase <- "t1_pre"
id <- grep(deletephrase, dir(delete_dir))
todelete <- dir(delete_dir, full.names=T)[id]
todelete
# [1] "d:/ff/t1_pre/t1_pre$b_cnt.ff"     "d:/ff/t1_pre/t1_pre$b_rct.ff"     "d:/ff/t1_pre/t1_pre$b_weit.ff"   
# [4] "d:/ff/t1_pre/t1_pre$deal_id.ff"   "d:/ff/t1_pre/t1_pre$m_id.ff"      "d:/ff/t1_pre/t1_pre$saled_qty.ff"

unlink(todelete)  #only delete .rdata and .rprofile

file.remove(todelete)
# [1] FALSE FALSE FALSE FALSE FALSE FALSE
# Warning messages:
# 1: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_cnt.ff', reason 'Permission denied'
# 2: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_rct.ff', reason 'Permission denied'
# 3: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_weit.ff', reason 'Permission denied'
# 4: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$deal_id.ff', reason 'Permission denied'
# 5: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$m_id.ff', reason 'Permission denied'
# 6: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$saled_qty.ff', reason 'Permission denied'

ffdrop("d:/ff/t2_pre/t2_pre$c_cnt.ff")
# $RData
# d:/ff/t2_pre/t2_pre$c_cnt.ff.RData 
#                              FALSE 

# $ffData
# d:/ff/t2_pre/t2_pre$c_cnt.ff.ffData 
#                               FALSE 

# Warning messages:
# 1: In file.remove(imgfile) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$c_cnt.ff.RData', reason 'No such file or directory'
# 2: In file.remove(zipfile) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$c_cnt.ff.ffData', reason 'No such file or directory'

finalize(t2_pre)
# $m_id
# [1] FALSE

# $deal_id
# [1] FALSE

close(t2_pre)
# [1] FALSE

finalizer(t2_pre)
# Error in UseMethod("finalizer") : 
#   no applicable method for 'finalizer' applied to an object of class "ffdf"

delete(t2_pre)
# [1] FALSE
# Warning messages:
# 1: In file.remove(attr(physical, "filename")) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$m_id.ff', reason 'Permission denied'

Answer 1

You first need to close your ff files before you can do file.remove. Something like close(yourffdf) or close(yourffobject) and next file.remove(list.files(yourpath), recursive=TRUE, full.names=TRUE) will do

require(ff)
x <- as.ffdf(iris)
sapply(filename(x), file.remove) ## Will fail
close(x)
sapply(filename(x), file.remove) ## Works