转换为numpy数组需要多少内存和处理

Question

I'm working on files that will requires me to load integers into an array that is ~18million in length.

How much memory and processing time will it take to do something like

my_list = [123,231,90,20,...,92] #length is 18 million
new_list = np.array(my_list, type='int')

Would this be the computer simply creating a second array that has 18 million in length and duplicating or does numpy do something more complicated.

Answer 1

In this case "processing time" doesn't come into a great deal since it's mostly a matter of pointer lookups. Since you tagged the post big-O, this is going to be O(n) . When converting a Python list to a Numpy array it will reserve memory to store len(my_list) integers in the array--you can find out exactly how much memory this is by seeing what the default int type is on your Numpy and using the .itemsize attribute of the dtype object:

>>> np.dtype('int')
dtype('int64')
>>> np.dtype('int').itemsize
8

So this array will require 8 * len(my_list) bytes to store on top of the memory already used by your original my_list .

It will then need to loop over each item in the list, look up what type of Python object it is (remember, Python list s can be heterogeneous--there's no way to know a priori that every item in the list will be convertible to an integer). Numpy will then do its best to convert that Python object to a machine integer and store it in the list.

Depending on where these 18 million integers are coming from it may be desirable not to store them in a Python list in the first place, if at all possible. But saying anything more about that would require more detail in the question.