如何使用numpy和strides来提高循环的性能

Question

I'm new to numpy and just heard about strides, but I'm having trouble getting my head wrapped around how to use them. I have a small loop that is very slow and it seems like a good example that might work for strides.

def semivariogram( ds, band, lag ):
    width = ds.RasterXSize
    height = ds.RasterYSize
    data = band.ReadAsArray( 0, 0, width, height ).astype(np.float)
    #print 'w: {}, h: {}'.format(width, height)

    sumw = 0.0
    sumh = 0.0
    for i in range(width-lag):
        for j in range(height-lag):
            sumw += data[i+lag,j]-data[i,j]
            sumh += data[i,j+lag]-data[i,j]

    Nh2 = 2.0*(width-lag)*(height-lag)

    return [sumw/Nh2, sumh/Nh2, (sumw/Nh2+sumh/Nh2)/2.0]

The line:

data = band.ReadAsArray( 0, 0, width, height ).astype(np.float)

is reading one band of an image into an array of floats. And ds is a handle to an image like:

ds = gdal.Open('test.tif')

Answer 1

Ok, let's do this step by step.

You have:

sumw = 0.0
sumh = 0.0
for i in range(width-lag):
    for j in range(height-lag):
        sumw += data[i+lag,j]-data[i,j]
        sumh += data[i,j+lag]-data[i,j]

Let's split this into two loops for clarity

sumw = 0.0
for i in range(width-lag):
    for j in range(height-lag):
        sumw += data[i+lag,j]-data[i,j]
sumh = 0.0
for j in range(height-lag):
    for i in range(width-lag):
        sumh += data[i,j+lag]-data[i,j]

We can write data[i,j+lag] as data[:-lag,lag:][i,j] , and data[i,j] is data[:-lag,:-lag][i,j] (for our range of i and j , and assuming lag != 0). So our loops become:

sumw = 0.0
for i in range(width-lag):
    for j in range(height-lag):
        sumw += data[lag:,:-lag][i,j]-data[:-lag,:-lag][i,j]
sumh = 0.0
for j in range(height-lag):
    for i in range(width-lag):
        sumh += data[:-lag,lag:][i,j]-data[:-lag,:-lag][i,j]

But now we note that all our loops are just iterating over a simple index, [i,j] , so we can flatten them entirely:

sumw = (data[lag:,:-lag] - data[:-lag,:-lag]).sum()
sumh = (data[:-lag,lag:] - data[:-lag,:-lag]).sum()

This can be made even faster by spotting that the sum of the difference is the difference of the sums:

sumw = data[lag:,:-lag].sum() - data[:-lag,:-lag].sum()
sumh = data[:-lag,lag:].sum() - data[:-lag,:-lag].sum()

At this point, you can think about the overall task visually:

You want sumw as the sum over the green outline minus the sum over the red outline, and sumh as the sum over the blue outline minus the sum over the red outline.

Here, we can make one of two simplifications:

1. We note that most of the elements in the first sum are being removed from the second sum. In fact, the only elements for which this is not the case are [:lag] and [-lag:] . So this becomes:

    sumw = data[-lag:,:-lag].sum() - data[:lag,:-lag].sum() sumh = data[:-lag,-lag:].sum() - data[:-lag,:lag].sum() 

   Which will be faster provided that lag < len(data) / 2 . Here, we are setting sumw = green - (red + magenta) , and sumh = blue - (red + yellow)

2. We note that the second sum is repeated, so have

    sum_shared = data[:-lag,:-lag].sum() sumw = data[lag:,:-lag].sum() - sum_shared sumh = data[:-lag,lag:].sum() - sum_shared