Numpy 如何在向量化中排除数组？

Question

I have a function:

import numpy as np

t = np.array([['t1', 0],['t2',0],['t3',1],['t4',1]])
i = np.array(['t3', 't4'])

def myfunc(d, x): 
    return d[:,1][np.where(d[:,0] == x)]

vfunc = np.vectorize(myfunc, excluded=['d'])
vfunc(d=t,x=i)

Expected output is: array(['1', '1'], dtype='<U2')

Gives the error: ValueError: setting an array element with a sequence

I don't see why this doesn't work following exclusion argument in documentation: https://numpy.org/doc/stable/reference/generated/numpy.vectorize.html

Answer 1

As hpaulj said Know, test, don't guess.

It was bit difficult to pin point, but here it is.

Docs says By default, pyfunc is assumed to take scalars as input and output.

To remove this error add otypes argument, you will not get your exact output but numpy can atleast figure out correctly what it needs to return

vfunc = np.vectorize(myfunc, excluded=['d'], otypes='O')

signature : string, optional

    Generalized universal function signature, e.g., (m,n),(n)->(m) for
 vectorized matrix-vector multiplication. If provided, pyfunc will be called
 with (and expected to return) arrays with shapes given by the size of
 corresponding core dimensions. By default, pyfunc is assumed to take scalars
 as input and output.

Below is detailed reason for proposed solution

Your function does not return a scalar output, it returns a vector output for each input scalar. While input is a scalar (t3,) and (t4,), there actual output are vector array(['1'], dtype='<U2') , array(['1'], dtype='<U2')] respectively.

Since you don't specify the signature, by default numpy thinks outputs are scalar and tries to put them in numpy array with dtypes as dtype of input(But actual dtype is object), trying to make a vector.

That is the error ValueError: setting an array element with a sequence , because your ufunc output is vector for each input scalar and not a scalar and also no signature is defined(reason for why numpy assumes output is scalar).

Also look at example similar to yours in numpy vectorize docs, numpy uses scalar return type and not vector

def mypolyval(p, x):

    _p = list(p)

    res = _p.pop(0)

    while _p:

        res = res*x + _p.pop(0)

    return res

vpolyval = np.vectorize(mypolyval, excluded=['p'])

vpolyval(p=[1, 2, 3], x=[0, 1])
array([3, 6])

I come to this conclusion through debugging, since myself don't have enough experience using vectorization.

While debugging in function_base.py , at line L2257 outputs are accumulated and has value array([array(['1'], dtype='<U2'), array(['1'], dtype='<U2')], dtype=object) . Then at L2260 numpy tries to convert them into required dtype, but fails because it was assuming sequence of scalars, but got a sequence of sequence.

Just put a breakpoint in vscode and try to see variable outputs you can figure out.

Answer 2

np.vectorize does "vectorize" in the sense that it allows you to pass array(s) to a function that otherwise only works with scalar values. But it can be tricky to use right, and it does not improve performance. It does not compile your function, so low level concepts of 'vectorization' do not apply.

---

In [1]: t = np.array([['t1', 0],['t2',0],['t3',1],['t4',1]])
   ...: i = np.array(['t3', 't4'])
   ...: 
In [2]: def myfunc(d, x):
   ...:     return d[:,1][np.where(d[:,0] == x)]
   ...: 
   ...: vfunc = np.vectorize(myfunc, excluded=['d'])

Your problem - with full traceback

In [3]: vfunc(d=t,x=i)
Traceback (most recent call last):
  File "<ipython-input-3-ea7904300378>", line 1, in <module>
    vfunc(d=t,x=i)
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py", line 2163, in __call__
    return self._vectorize_call(func=func, args=vargs)
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py", line 2249, in _vectorize_call
    res = asanyarray(outputs, dtype=otypes[0])
ValueError: setting an array element with a sequence

What does your function produce? arrays!

In [4]: myfunc(t,i[0])
Out[4]: array(['1'], dtype='<U21')
In [5]: myfunc(t,i[1])
Out[5]: array(['1'], dtype='<U21')

Lets try again, this time displaying the values passed to the function, and setting otypes . Since you read enough to use exclude you must have come across the otypes parameter as well. This issue often causes problems for SO questioners. (Based on the trial calculation your vectorize had set otypes to str , resulting in the ValueError )

In [6]: def myfunc(d, x):
   ...:     print(d,x)
   ...:     return d[:,1][np.where(d[:,0] == x)]
   ...: 
   ...: vfunc = np.vectorize(myfunc, excluded=['d'], otypes=['O'])
In [7]: vfunc(d=t,x=i)
[['t1' '0']
 ['t2' '0']
 ['t3' '1']
 ['t4' '1']] t3
[['t1' '0']
 ['t2' '0']
 ['t3' '1']
 ['t4' '1']] t4
Out[7]: 
array([array(['1'], dtype='<U21'), array(['1'], dtype='<U21')],
      dtype=object)

In [8]: np.hstack(_)
Out[8]: array(['1', '1'], dtype='<U21')

excluded did work, the whole d was passed each time. An alternative would be to define the function to use a global array, t , rather than expect it as argument

def myfunc(x):
   print(x)
   return t[:,1][np.where(t[:,0] == x)]

Let's generalize the arguments a bit

In [12]: t = np.array([['t1', 0],['t2',0],['t3',1],['t4',1],['t4',10]])
    ...: i = np.array(['t3', 't4','t5'])
    ...: 
    ...: 
In [13]: vfunc(d=t,x=i)
[['t1' '0']
 ['t2' '0']
 ['t3' '1']
...
Out[13]: 
array([array(['1'], dtype='<U21'), array(['1', '10'], dtype='<U21'),
       array([], dtype='<U21')], dtype=object)

the result is 3 arrays with different sizes. hstack would still work.

But with just one 1d argument, i , it would be just as easy, and faster, to use:

In [14]: np.array([myfunc(t,j) for j in i])
[['t1' '0']
 ['t2' '0']
 ['t3' '1']
 ....
<ipython-input-14-207a42bdb1fb>:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  np.array([myfunc(t,j) for j in i])
Out[14]: 
array([array(['1'], dtype='<U21'), array(['1', '10'], dtype='<U21'),
       array([], dtype='<U21')], dtype=object)

vectorization

This can have many meanings. simd vectorization is a new one for me, but then I haven't done low level ('C') programming in years. np.vectorize does not compile or alter your function in anyway. It has a clear performance disclaimer (I suppose it could be in bold and at the start of the docs). It's intended primarily for functions that take scalar values, and return similarly simple results, and is most useful when it takes several arguments, and for which you want to take advantage of numpy broadcasting .

The vectorized function evaluates `pyfunc` over successive tuples
of the input arrays like the python map function, except it uses the
broadcasting rules of numpy.

Informally (and without documented justification) we talk about numpy 'vectorization' as a way of making big performance gains. What that really means is moving iterations from the Python level (for loops, list comprehensions) to compiled numpy methods. It's interpreter vs compiler difference. np.vectorize , despite the name, does not do this.

A way of using numpy methods is:

In [17]: t[:,0]==i[:,None]
Out[17]: 
array([[False, False,  True, False, False],
       [False, False, False,  True,  True],
       [False, False, False, False, False]])

By adding a dimension to i , we can test t[:,0] against all i values at once. Applying where to that to get the indices:

In [19]: np.where(t[:,0]==i[:,None])
Out[19]: (array([0, 1, 1]), array([2, 3, 4]))

and using that to index t :

In [20]: t[_[1],1]
Out[20]: array(['1', '1', '10'], dtype='<U21')

Or we could use the boolean mask in Out[17] row by row:

In [21]: [t[j,1] for j in _17]
Out[21]: 
[array(['1'], dtype='<U21'),
 array(['1', '10'], dtype='<U21'),
 array([], dtype='<U21')]

With the possible mix of array sizes, it is hard to do this without some sort of python level iteration. The fast numpy operations work on multidimensional arrays, not on "ragged" ones.

Answer 3

I could be missing something, but the other answers seem to be really overcomplicating this. You just need 'd' to be a keyword argument, as the vectorize method seems to only be capable of interfering with the keywords dictionary.

I was dealing with this problem myself and switching the argument to a keyword fixed it instantly.