如何将 append 原语列表转换为 numpy 对象数组

Question

EDIT: I've gotten a lot of useful feedback on how not to do this and how to find alternatives, but making that useful depends on idiosyncrasies of my use case that would make this question less useful to others. At this point, I'm not looking for alternatives to using data structured like this . I'm looking for why it seems to be impossible to do this in numpy (or how to do it if it's not impossible)

I have a numpy array, which looks like

a = array([list([1]), list([4, 5])], dtype=object)

I want to append a list like

b = [2, 3, 4]

To get a result like

array([list([1]), list([4, 5]), list([2, 3, 4])], dtype=object)

However, every method I've tried has produced:

array([list([1]), list([4, 5]), 2, 3, 4], dtype=object)

I've tried vstack, concatenate, and append, as well as wrapping things in lists or ndarrays.

Why am I doing this? Basically, I have a lot of data in an ndarray that's going to get fed into sklearn. I want to have a 3d ndarray (data sets x data points x features) but incoming data is bad and certain things have different lengths, so the innermost dimension has to be lists. I'm trying to append a derived feature, which keeps failing. I've managed to reorder the operations to avoid needing to do this appending, but I still want to know how to do it. This seems like an odd failure for numpy. edit: In short, the outer array must be an ndarray because it's actually 2d, and complex slicing is frequently used, while the append operation occurs very few times.

Answer 1

Appending to an array in the first place is an expensive and generally smelly operation. The thing is that the contents of the array may be mutable, but the address of the underlying buffer is not. Every time you append an element, the whole thing gets reallocated and copied. As far as I'm aware, there isn't even an attempt at amortization, as with list .

If you are up for a slightly different approach, I would recommend maintaining your data in a list as you have now. You just transform your list into an array whenever you actually need the array. Remember that this is cheaper than reallocating to a new array every time, and you probably won't have to do it often compared to the number of appends:

stack = [[1], [4, 5]]
a = np.array(stack, dtype=np.object)
# do stuff to the array

...

stack.append([2, 3, 4])
a = np.array(stack, dtype=np.object)

Update Now that I Understand Your Question

If your goal is just to figure out how to append an element to an object array without having the fact that it is a list get in your way, you have to first create an array or element that is empty. Rather than trying to coerce the type with fake elements as some of the comments suggest, I recommend just creating empty elements and setting them to your list explicitly. You can wrap the operation in a function if you want to have a clean interface.

Here is an example:

b = [2, 3, 4]
c = np.empty(1, dtype=np.object)
c[0] = b
a = np.concatenate((a, c))

OR

a = np.append(a, c)

Of course this is not as clean as np.array([b], dtype=np.object) , but that's just an artifact of how numpy processes arrays. The reason you pretty much have to do it like this is that numpy treats anything that is a list or tuple as a special item that you want to convert into an array at the outer level.

Answer 2

Time passed, but maybe someone will make use of that (Python 3.9, NumPy 1.23).

I've had the same problem. The easiest solution I've found is to append one element to an ndarray (as a placeholder, in other words as an array extender), then assign the list to the last element of the extended array.

a_list = [1, 2, 3]
an_array = np.ones(10, dtype=object)
an_array = np.append(an_array, 0)
an_array[-1] = a_list

I think it has the smallest performance impact because the temporary array isn't created.

EDIT: I saw that JE_Muc's solution is almost the same as mine.

Answer 3

If you really must have an np.ndarray with dtype=object , you can do this:

a = np.array([list([1]), list([4, 5])], dtype=object)
b = [2, 3, 4]
a = np.hstack((a, np.empty(1)))
a[-1] = b

(Or of course remove np. in your case where you fully imported numpy.)
But I recommend not using np.ndarray s of dtype=object . Instead use list s with:

a = [[list([1]), list([4, 5])]]
b = [2, 3, 4]
a.append(b)

Now if you really want to have a as an np.ndarray , you can then do the following:

a = np.array(a)