无法为形状为 (129213603, 28) 且数据类型为 int8 的数组分配 3.37 GiB

Question

if I run my python Code on Spyder, i get the above error code. I have some pretty big lists, which I give into a multiindex

4 list:

customer_list = []
for y in df_1['Sold-to SAP Cust No']:
    if y not in customer_list:
        customer_list.append(y)
        customer_list.sort()


df_c = pd.MultiIndex.from_product([
    customer_list,
    Material_list,
    Week_list,
    base_list],

The lists have about 200-300 entries.

Does the error code mean that the lists are too big?

Answer 1

The issue is your attempting to allocate more memory than is available.

On Linux you can overcommit memory to a certain extent and on windows you can manage virtual memory.

Here is a link to a previous question on Stack Overflow that might be of some use. Unable to allocate array with shape and data type .

There are also some really interesting articles out there about managing memory. My honest suggestion would be to research ways to manipulate your data without putting it all in memory. You can store you data in a file or work with your data in a database. Here is an article which talks about large datasets, How to handle large datasets . The article is directed towards pandas, but the concepts are the same.

You will benefit in the long run, especially if you attempt to run this program on different systems or work with larger datasets.