将函数应用于列表的numpy数组中的每个列表

Question

I have a function that accepts a list (of strings). It does some processing on that list and returns another list of strings, possibly of shorter length.

Now, I have a numpy array of input lists of strings. I want to apply this transformation function to each list in my array.

From what searching I have done so far, it seemed like vectorize or apply_along_axis might be good candidates, but neither is working as expected.

I'd like to do this as efficiently as possible. Ultimately the input array will contain on the order of 100K lists.

I suppose I could iterate over the numpy array in a for loop, then append each output list into a new output array one at a time, but that seems horribly inefficient.

Here is what I've tried. For testing purposes, I've made a dumbed down transformation function and the input array contains just 3 lists.

def my_func(l):
    # accepts list, returns another list
    # dumbed down list transformation function
    # for testing, just return the first 2 elems of original list
    return l[0:2]

test_arr = np.array([['the', 'quick', 'brown', 'fox'], ['lorem', 'ipsum'], ['this', 'is', 'a', 'test']])

np.apply_along_axis(my_func, 0, test_arr)
Out[51]: array([['the', 'quick', 'brown', 'fox'], ['lorem', 'ipsum']], dtype=object)

# Rather than applying item by item, this returns the first 2 elements of the entire outer array!!

# Expected:
# array([['the', 'quick'], ['lorem', 'ipsum'], ['this', 'is']])

# Attempt 2...

my_func_vec = np.vectorize(my_func)
my_func_vec(test_arr)

Result:

Traceback (most recent call last):

  File "<ipython-input-56-f9bbacee645c>", line 1, in <module>
    my_func_vec(test_arr)

  File "C:\Users\Tony\Anaconda2\lib\site-packages\numpy\lib\function_base.py", line 2218, in __call__
    return self._vectorize_call(func=func, args=vargs)

  File "C:\Users\Tony\Anaconda2\lib\site-packages\numpy\lib\function_base.py", line 2291, in _vectorize_call
    copy=False, subok=True, dtype=otypes[0])

ValueError: cannot set an array element with a sequence

Answer 1

From the docstring of vectorize it reads about the optional argument otypes

otypes : str or list of dtypes, optional
    The output data type. It must be specified as either a string of
    typecode characters or a list of data type specifiers. There should
    be one data type specifier for each output.

It allows you to create structured arrays having complex output, but also solves your problem where you have lists as array element.

my_func_vec = np.vectorize(my_func, otypes=[list])

Answer 2

Some comparisons and time tests; but keep in mind that is a small example.

In [106]: test_arr = np.array([['the', 'quick', 'brown', 'fox'], ['lorem', 'ipsum'], ['this', 'is', 'a', 'test']])
     ...: 
In [107]: def my_func(l):
     ...:     # accepts list, returns another list
     ...:     # dumbed down list transformation function
     ...:     # for testing, just return the first 2 elems of original list
     ...:     return l[0:2]
     ...: 

The list comprehension method returns a 2d array of strings - because the function returns 2 element lists each time.

In [108]: np.array([my_func(x) for x in test_arr])
Out[108]: 
array([['the', 'quick'],
       ['lorem', 'ipsum'],
       ['this', 'is']],
      dtype='<U5')

The input array is object dtype because the sublists differ in length:

In [109]: test_arr
Out[109]: 
array([list(['the', 'quick', 'brown', 'fox']), list(['lorem', 'ipsum']),
       list(['this', 'is', 'a', 'test'])], dtype=object)

frompyfunc returns an object dtype array; consistent with my past tests it is modestly faster (2x but never an order of magnitude)

In [110]: np.frompyfunc(my_func,1,1)(test_arr)
Out[110]: 
array([list(['the', 'quick']), list(['lorem', 'ipsum']),
       list(['this', 'is'])], dtype=object)

In [111]: timeit np.frompyfunc(my_func,1,1)(test_arr)
5.68 µs ± 230 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [112]: timeit np.array([my_func(x) for x in test_arr])
8.96 µs ± 25.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

vectorize uses frompyfunc but has more overhead. The otypes is need to avoid the sequence error (otherwise it tries to deduce the return type from a trial calculation):

In [113]: np.vectorize(my_func,otypes=[object])(test_arr)
Out[113]: 
array([list(['the', 'quick']), list(['lorem', 'ipsum']),
       list(['this', 'is'])], dtype=object)
In [114]: timeit np.vectorize(my_func,otypes=[object])(test_arr)
30.4 µs ± 132 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Answer 3

[my_func(x) for x in test_arr]

您需要向下移动一级，您的解决方案仅输出阵列的前2个项目，而不是阵列的每个前2个项目。