为什么 numpy 数组比带有 for 循环的列表慢？

Question

Aren't arrays supposed to be faster since they consume less memory and as I know with arrays python doesn't apply type method on the elements as it in the lists.

import numpy as np
import time


length = 150000000

my_list = range(length)


list_start_time = time.time()


for item in my_list:
    pass

print(f'my_list finished in: {time.time() - list_start_time}')
# # Output => my_list finished in: 3.57804799079895

my_array = np.arange(length)

array_start_time = time.time()


for item in my_array:
    pass

print(f'my_array finished in: {time.time() - array_start_time}')
# # Output => my_array finished in: 11.598113536834717

Answer 1

my_list = range(length) is a range object, more of a generator than a list

In the loop:

 for i in range(10):
      pass

there's no significant memory use. But even if we did iterate on a list, each i would just be a reference to an item in the list. In effect a simple pointer. The list has a data buffer, which contains pointers to objects elsewhere in memory. Iteration simply requires fetching those pointers, without any object creation or processing.

In arr = np.arange(10) , arr is an array object with a datebuffer containing bytes representing the values of the integers, 8 bytes per item (in the default dtype).

 for i in arr:
      pass

numpy indexes each element, fetching the relevant 8 bytes (relatively fast), and converting them to a number. The whole process is more involved than simply fetching a reference from a list's data buffer. This process is sometimes called 'unboxing'.

To illustrate, make alist and array from that list:

In [4]: alist = list(range(1000))
In [5]: arr = np.array(alist)

Indexing the list returns a python int object; from the array we get a numpy object:

In [6]: type(alist[0])
Out[6]: int
In [7]: type(arr[0])
Out[7]: numpy.int64

Some timings:

In [8]: timeit [i for i in alist]
27.9 µs ± 889 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [9]: timeit [i for i in arr]
124 µs ± 625 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

iteration on the list is much faster (as you note); and based on the following timing it looks like the array iteration effectively does [i for i in list(arr)] :

In [10]: timeit list(arr)
98 µs ± 661 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

The tolist method converts the array to a list, all the way down (to native elements), and is much faster. [i for i in arr.tolist()] will actually save time.

In [11]: timeit arr.tolist()
22.8 µs ± 28 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Another way to illustrate the 'unboxing' is to look at the id of the same element (taking care to avoid memory reuse):

In [13]: x, y = alist[10], alist[10]; id(x), id(y)
Out[13]: (10914784, 10914784)
In [14]: x, y = arr[10], arr[10]; id(x), id(y)
Out[14]: (140147220887808, 140147220887832)

Each time we index a list element, we get the same id , the same object.

Each time we index an array element, we get a new object. That object creation takes time.

numpy arrays are faster - if we do the iteration is compiled c code.

For example to add 100 to each element of the array or list:

In [17]: timeit arr + 100
3.46 µs ± 136 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [18]: timeit [i+100 for i in alist]
60.1 µs ± 125 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Answer 2

np.arrays are much faster when using vectorized functions or to say it another way, specifically when you are not iterating over the elements.

Here is an example. We will add 1 to each number in the ordered list you created.

The setup is similar except I use lists instead of ranges so I can operate on the values.

length = 150000000
my_list = list(range(length))
my_array = np.array(my_list)

Using the %%timeit magic function in jupyter notebooks

%%timeit
for i in range(len(my_list)):
    my_list[i] += 1

gives

38.6 s ± 3.71 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

thats about 38 seconds per loop

On the other hand

%%timeit
new_array = my_array + 1

gives

772 ms ± 58.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The operation on the numpy array took under one second (~772ms) per loop while iterating through the list and adding one to each element too around 36 seconds per loop.