Matlab：过滤大型数组元素，更快地替代逻辑索引吗？

Question

I have a large, three-dimensional dataset of floats, roughly 500 million elements (3000 x 300 x 600).

I want to make elements that are below or above certain thresholds zero. Logical indexing can do this, eg

cut_in = 0.5
cut_out = 6
Hs(Hs<cut_in) = 0 ;
Hs(Hs>cut_out) = 0 ;

The problem is that this is painfully slow for me, what with the large data size. The above code takes 240 seconds to run on my computer. Is there a quicker way I can do this?

Many thanks

Answer 1

As @rayryeng and @AndrasDeak point out in comments to your question, logical indexing is usually fastest, though your runtimes suggest that you are probably limited by memory (and being forced to swap onto disk) rather than by the actual speed of the indexing.

One surprising alternative that can win out in this case is for loops. This is because logical indexing requires three passes through the array (once for each inequality test, and once to change the data), whereas a for loop only requires one pass through the array.

Benchmarks

So I ran these tests (and accidentally doubled the array size) on a machine with 8 GB memory:

>> A = randn(6000,300,600);
>> cut_in = -1;
>> cut_out = 1;

Using a for loop:

>> tic; for i=1:numel(A), if A(i)<cut_in || A(i)>cut_out, A(i)=0; end; end; toc
Elapsed time is 597.384884 seconds.

Using logical indexing:

>> tic; A(A<cut_in | A>cut_out) = 0; toc
Elapsed time is 1619.105332 seconds.

And just for laughs (had some time on my hands waiting for the benchmarks to run), here is a compiled for loop:

#include "mex.h"

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
    double *A = mxGetPr(prhs[0]);
    size_t N = mxGetNumberOfElements(prhs[0]);
    double cut_in = *mxGetPr(prhs[1]);
    double cut_out = *mxGetPr(prhs[2]);
    // You're not supposed to do in-place operations! Don't do this!
    for (ptrdiff_t ii=0; ii<N; ii++) {
        if ((A[ii]<cut_in) || (A[ii]>cut_out))
            A[ii] = 0;
    }
}

And benchmarked:

>> mex -v CXXOPTIMFLAGS="-O3 -DNDEBUG" -largeArrayDims apply_threshold.cpp
>> tic; apply_threshold(A,cut_in,cut_out); toc
Elapsed time is 529.994643 seconds

One thing to keep in mind is that we are operating in a regime where accessing the swap space is the main bottleneck in performance, and so benchmark results can vary, even within the same machine, depending on what's currently in main memory (vs. what needs to be swapped in) and what kind of background processes are running.