Dictionary of 4-D numpy array: how to optimize?

Question

I have a problem of performances in inizializing a dictionary of 4-D numpy tensor.

I have a list of coefficients names:

cnames = ['CN', 'CM', 'CA', 'CY', 'CLN' ...];

that is not fixed-size (it depends from upper code). For each coefficient i have to generate a 4-D tensor [nalpha X nmach X nbeta X nalt] of zeros (for preallocation purposes), so I do:

#Number of coefficients
numofc = len(cnames);

final_data = {};
#I have to generate <numofc> 4D matrixes
for i in range(numofc):
    final_data[cnames[i]]=n.zeros((nalpha,nmach,nbeta,nalt));

each index is an integer between 10 and 30. each index is an integer between 100 and 200

This takes like 4 minutes. How can I speed up this? Or am I doing something wrong?

Answer 1

The code you posted should not take 4 minutes to run (unless cnames is extremely large or you have very little RAM and is forced to use swap space).

import numpy as np

cnames = ['CN', 'CM', 'CA', 'CY', 'CLN']*1000
nalpha,nmach,nbeta,nalt = 10,20,30,40
#Number of coefficients
numofc = len(cnames)

final_data = {}
#I have to generate <numofc> 4D matrixes
for i in range(numofc):
    final_data[cnames[i]]=np.zeros((nalpha,nmach,nbeta,nalt))

Even if cnames has 5000 elements, it should still take only on the order of a couple seconds:

% time test.py
real    0m4.559s
user    0m0.856s
sys 0m3.328s

The semicolons at the end of statements suggests you have experience in some other language. Be careful of translating commands from that language line-by-line into NumPy/Python. Coding in NumPy as one would in C is a recipe for slowness.

In particular, try to avoid updating elements in an array element-by-element. This works fine in C, but is very slow with Python. NumPy achieves speed by delegating to functions coded in Fortran or Cython or C or C++. By updating arrays element-by-element you are using Python loops which are not as fast.

Instead, try to rephrase your computation in terms of operations on whole arrays (or at least, slices of arrays).

I have probably speculated too much on the cause of the problem. You need to profile your code , and then, if you want more specific help, post the result of the profile plus the problematic code (most helpfully in the form of an SSCCE ).