带有重叠切片的NumPy就地操作

Question

Consider this out-of-place operation:

>>> b = numpy.asarray([1, 2, 3])
>>> b[1:] = b[1:] - b[:-1]
>>> b
array([1, 1, 1])

Now consider the in-place operation:

>>> a = numpy.asarray([1, 2, 3])
>>> a[1:] -= a[:-1]
>>> a
array([1, 1, 2])

They give different results, which I did not expect.

Update: It seems this behavior changed; now they give the same result.

I would have assumed NumPy would have done the subtraction in the correct order (backward) so that they would give equivalent results to the out-of-place subtraction.

My question is: is this intended behavior on NumPy's part, or is it a bug, or is the result undefined?

Answer 1

Undefined, or at least hard-to-understand, may be best answer. Another SO answer claims

a[1:] -= a[:-1]

gets translated, by the interpreter, into something like

a.__setitem__(slice(1,None), a.__getitem__(slice(1,None)).
   __isub__(a.__getitem__(slice(None,-1))))

In [171]: a=np.arange(10)
In [172]: a[1:] -= a[:-1]
In [173]: a
Out[173]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])

evaluates the same as:

In [175]: for i in range(9):
     ...:     a[i+1] = a[i+1]-a[i]

I can sort of see how it gets this from the nested __setitem__ etc expression. I'm try to replicate it with np.nditer .

the reverse that you mention would be

In [178]: for i in range(8,-1,-1):
     ...:     a[i+1] = a[i+1]-a[i]

I don't there's any way that numpy could deduce that such a reverse iteration is required. The 2nd argument to __setitem__ evaluates just fine with forward iteration. Buffering that term is the only simple solution.

The .at ufunc method was introduced as a way to get around buffering issues in expressions like a[idx] += b . In particular when idx has dupicates. Should the effect on a be cumulative or should just the last instance apply.

In your example if behaves just like a[1:] - a[:-1] :

In [165]: a=np.arange(10)
In [166]: idx=np.arange(1,10)
In [167]: np.add.at(a, idx, -a[:-1])
In [168]: a
Out[168]: array([0, 1, 1, 1, 1, 1, 1, 1, 1, 1])

That's because 3rd argument to add.at is fully evaluated before being used. It's a temporary copy. I know from other tests that add.at is slower than the ordinary a[idx] += . [I'm a little confused as to what 'buffering' add.at is bypassing; how that's different from the apparent lack of buffering that is giving problems here?]

Why use the += notation? Just to make the code more compact? or in hopes of making it faster? But if speed is the goal, do we want numpy throw in some extra buffering, just to make it safer?

---

An nditer equivalent of the a[1:] -= a[:-1]

In [190]: a=np.arange(10)
In [191]: it = np.nditer([a[1:],a[1:],a[:-1]], op_flags=['readwrite'])
In [192]: for i,j,k in it:
     ...:     print(i,j,k)
     ...:     i[...] = j-k
     ...:     print(i)

1 1 0
1
2 2 1
1
3 3 1
2
4 4 2
2
5 5 2
3
6 6 3
...

The iteration can be simplified

In [197]: it = np.nditer([a[1:],a[:-1]], op_flags=['readwrite'])
In [198]: for i,j in it:
     ...:     i[...] -= j

Because it's a view the iterated value from a[:-1] will reflect changes made in the previous loop.

I'm not sure that c version of nditer is being using in the array += expression, but the intention of nditer was to consolidate the iteration coding into one unified framework.

---

Another interesting observation is that if I define

idx = array([1, 2, 3, 4, 5, 6, 7, 8, 9])

then

a[idx] -= a[:-1]
a[1:] -= a[idx-1]
a[idx] -= a[idx-1]

all give the desired array([0, 1, 1, 1, 1, 1, 1, 1, 1, 1]) . In other words both sides of the -= have to be 'views/slices'. That must be the buffering that add.at bypasses. That a[idx-1] is a copy is obvious. That a[idx]-= throws in a buffer, while a[1:]-= does not, is not so obvious.

Answer 2

This behavior was previously undefined, but since NumPy 1.13.0 , operations with overlapping input and output now behave as if inputs were copied first. Quoting the release notes:

Operations where ufunc input and output operands have memory overlap produced undefined results in previous NumPy versions, due to data dependency issues. In NumPy 1.13.0, results from such operations are now defined to be the same as for equivalent operations where there is no memory overlap.

Operations affected now make temporary copies, as needed to eliminate data dependency. As detecting these cases is computationally expensive, a heuristic is used, which may in rare cases result to needless temporary copies. For operations where the data dependency is simple enough for the heuristic to analyze, temporary copies will not be made even if the arrays overlap, if it can be deduced copies are not necessary. As an example, np.add(a, b, out=a) will not involve copies.

Here is the relevant issue on GitHub .