how to reduce memory demands of Matlab 'find' function?

Question

I have a line of code in Matlab that reads:

  output = find(input);

where column vector "output" contains all indices in column vector "input" whose elements are nonzero. For example, if:

 input = [1 3 4 0 0 2 0];

then the result of, output = find(input); would be:

 output = 
     1
     2
     3
     6

corresponding to the 1st ("1"), 2nd ("3"), 3rd ("4"), and 6th ("2") indices of array "input" that are nonzero.

Since my "input" array is very large, this line of code consumes all of my local RAM plus a huge portion of virtual memory, causing the system to slow to a crawl.

Anyone know of a good way (or any way) to reduce the memory requirements of such an operation? I thought about putting the "find" code in a loop, but since the size of array "output" (and thus indexing of this array) depends on the result of the "find" operation, I don't see how it's possible to do so. Ran out of ideas.

Thanks in advance for any comments/suggestions. -gkk

Answer 1

If you have more non-zero values than zeros, maybe you can work with the complement, ie: output=find(input==0) instead of the default which is equivalent to output=find(input~=0)

Also, you can use logical indexing, compare:

>> output1 = find(input);
>> output2 = (input~=0);
>> whos output*
  Name         Size            Bytes  Class      Attributes

  output1      1x4                32  double               
  output2      1x7                 7  logical    

note how it is stored as vector of "booleans", which is one byte each (vs 8 bytes for "double")

Answer 2

If you have enough RAM to hold an array the same size of input , you can replace the call to find by

output = 1:length(input);
output = output(input~=0);

If input has less than 2^32-1 elements, you can initialize it as uint32 , and thus further save on memory.

A better way might be to convert your input array to sparse , which saves memory if input contains lots of zeros, and then use find on that, ie

input = sparse(input);
output = find(input);

---

EDIT

To perform the find operation in pieces, I'd do the following:

nIn = length(input);
blockLength = 100000;
nBlocks = ceil(nIn/blockLength); %# work in chunks of 100k entries
out = cell(nBlocks,1);
for i=1:nBlocks
   out{i} = (i-1)*blockLength+1:i*blockLength; %# assign as your favorite integer format here
   out{i} = out{i}(input(out{i})~=0);
end
out = cat(1,out{:});