从列表和数组中过滤空字符串

Question

I have a list with empty strings:

test = ['foo', '', 'bar', '', 'baz']

The following code will strip the empty strings and return the desired output:

list(filter(None, test))
Out:['foo', 'bar', 'baz']

When I turn the list into a numpy array, applying the same function by mapping does not work:

test = np.array(['foo', '', 'bar', '', 'baz'], dtype='<U15')

def g(x):
    return list(filter(None, x))

def array_map(x):
    return np.array(list(map(g, x)))

array_map(test)
Out: array([list(['f', 'o', 'o']), list([]), list(['b', 'a', 'r']), list([]),
       list(['b', 'a', 'z'])], dtype=object)

Why does this happen and what is the correct, simple method to remove empty strings from a numpy array?

Answer 1

When I turn the list into a numpy array, applying the same function by mapping does not work

Right; the function already turns your source sequence into the list that you want to make an array out of , so there is no reason to do any mapping.

Why does this happen

Mapping g onto test means that g is separately called with each element of x . The elements of test are strings; when list(filter(None, x)) is evaluated with x being one of the strings from test , filter iterates over the characters of the string. All of those characters pass the filter, so a list is made that contains them. The map ped version of test , therefore, contains a bunch of lists of characters, which then is passed to np.array .

and what is the correct, simple method to remove empty strings from a numpy array?

Well, if you wanted to do it with filter , it would look like passing the Numpy array to a single call to filter , and then constructing a new array from the result. Only, the resulting filter object won't be iterated over automatically by np.array , so you'd have to create eg a list first. Thus:

>>> np.array(list(filter(None, test)), dtype='<U15')
array(['foo', 'bar', 'baz'], dtype='<U15')

(Notice that the dtype needs to be specified explicitly if you want it preserved; otherwise Numpy will infer the smallest type that suffices for the data.)

However, it is better to use Numpy tools for this task. The idiomatic way to remove data from an array is to create a mask that matches the elements you want, and index with that:

>>> test[test != '']
array(['foo', 'bar', 'baz'], dtype='<U15')

(If you want to remove everything that's false-ish - ie that would fail to satisfy an if condition - you can use the somewhat awkwardly named nonzero method: test[test.nonzero()] .)

Answer 2

In [714]: test = ['foo', '', 'bar', '', 'baz']                                                       

I like the expressiveness of list comprehensions:

In [715]: [s for s in test if s]                                                                     
Out[715]: ['foo', 'bar', 'baz']

This comprehension also works with an array - though it will be slower:

In [716]: aTest=np.array(test)                                                                       
In [717]: aTest                                                                                      
Out[717]: array(['foo', '', 'bar', '', 'baz'], dtype='<U3')
In [718]: np.array([s for s in aTest if s])                                                          
Out[718]: array(['foo', 'bar', 'baz'], dtype='<U3')

An element of the array tests the same an element of a list.

filter acts the same way:

In [724]: list(filter(None, test))                                                                   
Out[724]: ['foo', 'bar', 'baz']
In [725]: list(filter(None, aTest))                                                                  
Out[725]: ['foo', 'bar', 'baz']

Your two function approach ends up applying list to each string, splitting it. The outer map passes a string to g , not the whole list:

In [728]: def g(x): 
     ...:     return list(filter(None,x))                                                                                      
In [729]: list(map(g,test))                                                                          
Out[729]: [['f', 'o', 'o'], [], ['b', 'a', 'r'], [], ['b', 'a', 'z']]
In [732]: [list(s) for s in test]                                                                    
Out[732]: [['f', 'o', 'o'], [], ['b', 'a', 'r'], [], ['b', 'a', 'z']]
In [734]: list(g(test[0]))                                                                           
Out[734]: ['f', 'o', 'o']

As pointed out in the other answer, you can do the array filtering without a python level iteration:

In [736]: aTest==''                                                                                  
Out[736]: array([False,  True, False,  True, False])
In [737]: aTest[aTest!='']                                                                           
Out[737]: array(['foo', 'bar', 'baz'], dtype='<U3')

For this small sample the list comprehension is fastest. I expect though that with a 1000 string list/array, the array approach will scale better.

In [740]: timeit [s for s in test if s]                                                              
398 ns ± 10.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [741]: timeit aTest[aTest!='']                                                                    
3.99 µs ± 158 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [743]: timeit list(filter(None, test))                                                            
503 ns ± 9.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)