model.matrix raises memory allocation error

Question

I am using model.matrix to create many columns out of an existing data frame. The goal is to create many columns each of which will have a name equal to a distinct value of one feature column ( my_one_feature ). That is, if my_one_feature is a categorical variable with values {cat_1,cat_2,cat_3} , I expect 3 additional columns to be generated with name: cat_1 , cat_2 , cat_3 and the values for each will be 0 or 1 depending on their presence in corresponding row.

mm <- model.matrix(~factor(my_one_feature)-1,data=my_data_frame)

then I can

cbind(my_data_frame,mm)

I think the function task is exactly what I explained. However for large data and/or large feature values, it gives a memory allocation error:

cannot allocate vector of size 50 Gb

I know the resulting matrix will be sparse. How to avoid this memory allocation problem?

Here is an example with 4 original features with only 7 rows:

f1<-c('f1_1','f1_2','f1_1','f1_3','f1_3','f1_1','f1_4')
f2<-c(1,2,3,4,2,4,2)
f3<-c(1,2,3,4,5,6,7)
f4<-c(0,0,1,1,1,0,1)`

my_data_frame<-data.frame(f1,f2,f3,f4)

It looks like:

my_data_frame
    f1 f2 f3 f4
1 f1_1  1  1  0
2 f1_2  2  2  0
3 f1_1  3  3  1
4 f1_3  4  4  1
5 f1_3  2  5  1
6 f1_1  4  6  0
7 f1_4  2  7  1

mm<-sparse.model.matrix(~factor(f1)-1,data=my_data_frame)

looks like:

7 x 4 sparse Matrix of class "dgCMatrix"
  factor(f1)f1_1 factor(f1)f1_2 factor(f1)f1_3 factor(f1)f1_4
1              1              .              .              .
2              .              1              .              .
3              1              .              .              .
4              .              .              1              .
5              .              .              1              .
6              1              .              .              .
7              .              .              .              1

how to combine my_data_frame with mm such that resulting object can have all features columns (f1, f2, f3, f4, factor(f1)f1_1, factor(f1)f1_2, factor(f1)f1_3, factor(f1)f1_4)) and of course 7 rows.

Ok

your answer give this results on my rstudio tool:

> my_data_frame <- data.frame(
+     f1=c('f1_1','f1_2','f1_1','f1_3','f1_3','f1_1','f1_4'),
+     f2=c(1,2,3,4,2,4,2),
+     f3=c(1,2,3,4,5,6,7),
+     f4=c(0,0,1,1,1,0,1))
> library("Matrix")
> mm <- sparse.model.matrix(~factor(f1)-1,
+                           data=my_data_frame)
> new_data_frame <- cbind(Matrix(as.matrix(my_data_frame[,-1])),
+                         mm)
> dim(new_data_frame)
[1] 1 2
> str(new_data_frame)
List of 2
 $ :Formal class 'dgeMatrix' [package "Matrix"] with 4 slots
  .. ..@ x       : num [1:21] 1 2 3 4 2 4 2 1 2 3 ...
  .. ..@ Dim     : int [1:2] 7 3
  .. ..@ Dimnames:List of 2
  .. .. ..$ : NULL
  .. .. ..$ : chr [1:3] "f2" "f3" "f4"
  .. ..@ factors : list()
 $ :Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
  .. ..@ i       : int [1:7] 0 2 5 1 3 4 6
  .. ..@ p       : int [1:5] 0 3 4 6 7
  .. ..@ Dim     : int [1:2] 7 4
  .. ..@ Dimnames:List of 2
  .. .. ..$ : chr [1:7] "1" "2" "3" "4" ...
  .. .. ..$ : chr [1:4] "factor(f1)f1_1" "factor(f1)f1_2" "factor(f1)f1_3" "factor(f1)f1_4"
  .. ..@ x       : num [1:7] 1 1 1 1 1 1 1
  .. ..@ factors : list()
 - attr(*, "dim")= int [1:2] 1 2
 - attr(*, "dimnames")=List of 2
  ..$ : NULL
  ..$ : chr [1:2] "" "mm"
> 


> sessionInfo()
R version 3.1.3 (2015-03-09)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8 x64 (build 9200)

locale:
[1] LC_COLLATE=Lithuanian_Lithuania.1257  LC_CTYPE=Lithuanian_Lithuania.1257    LC_MONETARY=Lithuanian_Lithuania.1257 LC_NUMERIC=C                         
[5] LC_TIME=Lithuanian_Lithuania.1257    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Matrix_1.2-2

loaded via a namespace (and not attached):
[1] grid_3.1.3      lattice_0.20-30 tools_3.1.3    
> 

Answer 1

Set up data:

 my_data_frame <- data.frame(
 f1=c('f1_1','f1_2','f1_1','f1_3','f1_3','f1_1','f1_4'),
 f2=c(1,2,3,4,2,4,2),
 f3=c(1,2,3,4,5,6,7),
 f4=c(0,0,1,1,1,0,1))

Now use sparse.model.matrix for the categorical feature:

library("Matrix")
mm <- sparse.model.matrix(~factor(f1)-1,
           data=my_data_frame)

Bind it back to the numerical predictors (coerce data.frame -> matrix -> Matrix ):

new_data_frame <- cbind(Matrix(as.matrix(my_data_frame[,-1])),
                        mm)

Results:

dim(new_data_frame)
## [1] 7 7 
str(new_data_frame)
## Formal class 'dgeMatrix' [package "Matrix"] with 4 slots
##   ..@ x       : num [1:49] 1 2 3 4 2 4 2 1 2 3 ...
##   ..@ Dim     : int [1:2] 7 7
##   ..@ Dimnames:List of 2
##   .. ..$ : chr [1:7] "1" "2" "3" "4" ...
##   .. ..$ : chr [1:7] "f2" "f3" "f4" "factor(f1)f1_1" ...
##   ..@ factors : list()

object.size(new_data_frame) ## 1596 bytes

The result does not contain the original f1 column, because matrices can't have heterogeneous types -- but there would be no way to use that column its raw form for numerical modeling and prediction in any case ...

Session info (OP is using 3.1.3/windows 8 x64/Lithuanian locale/Matrix_1.2-2/tools_3.1.3):

R version 3.2.1 (2015-06-18)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.9.5 (Mavericks)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Matrix_1.2-2

loaded via a namespace (and not attached):
[1] grid_3.2.1      lattice_0.20-33