R: reshape/cast memory error

Question

I have a large table (x) to covert to matrix (y).

I used two different commands.

x <- reshape(y, direction="wide", v.names="column1", 
             timevar="column2", idvar="column3")

or

x <- cast(x, column1~column2)

After waiting for several hours, an error message appears.

 Error: cannot allocate vector of size 87 KB

In addition: There were 50 or more warnings (use warning() to see the first 50)

when I type warnings() , I found it is complainting

Reached total allocation of 1535 Mb:
see help(memory.size)

In a second trial, I type " memory.size(TRUE) first, a message comes up:

[1] 12.88

Then, I do the analysis all over again. However, the same error message appears.

I am using a desktop PC with windows 2000. I am using R2.12.0

Could you mind to let me know if it is possible to get rid of this problem?

Could you mind to let me know when I firstly type memory.size(TRUE) "12.88" comes up, does it mean that it all the 12G physical are RAM readily used by R in the second trial?

Answer 1

On most windows systems, memory usage is very limited (in R).

You want to look at the results of memory.limit instead of memory.size, and then also use memory.limit to alter it.

In the best case scenario, you may crank this up to maybe 4 Gb, which may still be too little for your purpose (depends on the details of your OS - apparently, your machine has enough RAM, so physical memory is not the issue here (yet)).

See rw-FAQ for more info.

Answer 2

Are you sure you have 12GB of RAM or that you are running a 64-bit version of Windows 2000? If you read ?memory.size you'll see that the value is in bytes not GB.

The main message states that you are running out of physical memory. The amount mentioned in the warnings of 1535MB is indicative of a machine with 2GB physical RAM or an OS that can only access that amount or allocate such an amount to a single process. If you really have 12GB physical RAM then you'll need to be running a 64-bit version of R on a 64-bit version of Windows on a 64-bit processor. Do all of these apply? It looks like for example that you might be running on a machine with 2GB of RAM or using a 32-bit version of R.

The problem boils down to exhausting the available RAM that R can access. Solve that or cut your problem down into smaller chunks and see if you can reshape each smaller chunk and then stick them together.