级联numpy logical_and导致内存错误

Question

For a GIS problem, I am using numpy within gdal_calc.py and run into immediate MemoryError. My goal is to compare multiple (up to 500) arrays of same length to see which array contains the highest value at a specific index - and return an array of same length with idcodes associated with the (first, in case there are multiple) "winning" array.

Suppose AA = [1,3,2,8,9] and AB = [4,1,3,9,7] and AC = [2,1,3,10,8] , the expected result is [idcodeAB,idcodeAA,idcodeAB,idcodeAC,idcodeAA]

This is my numpy code:

logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(AA>=AB,AA>=AC),AA>=AD),AA>=AE),AA>=AF),AA>=AG),AA>=AH),AA>=AI),AA>=AJ),AA>=AK),AA>=AL) * idcodeAA + 
logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(AB>=AA,AB>=AC),AB>=AD),AB>=AE),AB>=AF),AB>=AG),AB>=AH),AB>=AI),AB>=AJ),AB>=AK),AB>=AL) * idcodeAB + 
logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(logical_and(AC>=AA,AC>=AB),AC>=AD),AC>=AE),AC>=AF),AC>=AG),AC>=AH),AC>=AI),AC>=AJ),AC>=AK),AC>=AL) * idcodeAC

I hope my explanation and code example makes sense - it surely looks ugly (particularly when scaled up). And it creates immediate MemoryErrors. Can anybody suggest a nicer way to code this? Thanks in advance!

Answer 1

Why not np.vstack the arrays together and use argmax on each index? For example

>>> import numpy as np
>>> # 500 arrays of random numbers
>>> x = [np.random.rand(100) for _ in xrange(500)]
>>> y = np.vstack(x)
>>> y.shape
(500, 100)

>>> y.argmax(axis=0)
array([454,  94, 197,  47, 102, 359, 480, 432, 101, 383, 441, 358,  32,
       474, 476,  94, 454, 227, 327, 336, 302, 114, 368,  41, 362, 136,
       337, 439, 313, 259, 270, 132, 494, 264, 393, 264, 153, 290, 497,
       256, 231, 277, 455,  70, 288, 173, 499,  91, 256, 388, 284, 348,
       123, 482,  72, 153, 347, 113,  24, 141,  68, 440, 244, 113,  69,
        30, 472, 152, 106, 453,  13, 134, 169, 205, 317, 127, 248, 352,
        25, 445, 470, 167,  51, 183,  95, 462, 394, 491, 384, 150, 192,
       129, 407,  34, 249, 450, 398, 332, 142, 179])

>>> y.argmax(axis=0).shape
(100,)

So y.argmax(axis=0) gives you which array in y has the highest value at each index. For example, array 455 (454+1 due to 0 indexing) has the highest value at index 0 ( y[454,0] >= y[:,0] ).

Then you could do something like

np.array([y[i] * id for i, id in zip(y.argmax(axis=0), idcodes)]).sum(axis=0)

EDIT:

For example: with your edit:

In [24]: labels = 'ABC' #each label corresponds to one array in x
In [25]: x = [[1,3,2,8,9], [4,1,3,9,7], [2,1,3,10,8]]
In [26]: y = np.array(x)
In [27]: y.argmax(axis=0)
Out[27]: array([1, 0, 1, 2, 0])
In [28]: [labels[i] for i in y.argmax(axis=0)]
Out[28]: ['B', 'A', 'B', 'C', 'A']