Python raises Memory Error despite of 16gb Swap

Question

I'm getting MemoryError while running some large matrix operations (chroma, cqt, mfcc extraction) with numpy (1.81), scipy (0.17.0), librosa (0.4.2) on a Jetson TK 1 with ~2 GB Ram and a 16GB swap file.

Any help is much appreciated!

ERROR MESSAGE

    Traceback (most recent call last):
  File "./analyze_structure.py", line 480, in <module>
    args.cutoff, args.order, args.sr, args.feature, bool(args.as_diff))
  File "./analyze_structure.py", line 452, in plotData
    tracks)
  File "./analyze_structure.py", line 178, in plotStructure
    feat, beat_times = extractChroma(filename, file_ext)
  File "./analyze_structure.py", line 75, in extractChroma
    hop_length=HOP_LENGTH)
  File "/usr/local/lib/python2.7/dist-packages/librosa-0.4.2-py2.7.egg/librosa/feature/spectral.py", line 800, in chroma_stft
    tuning = estimate_tuning(S=S, sr=sr, bins_per_octave=n_chroma)
  File "/usr/local/lib/python2.7/dist-packages/librosa-0.4.2-py2.7.egg/librosa/core/pitch.py", line 82, in estimate_tuning
    pitch, mag = piptrack(y=y, sr=sr, S=S, n_fft=n_fft, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/librosa-0.4.2-py2.7.egg/librosa/core/pitch.py", line 270, in piptrack
    util.localmax(S * (S > (threshold * S.max(axis=0)))))
  File "/usr/local/lib/python2.7/dist-packages/librosa-0.4.2-py2.7.egg/librosa/util/utils.py", line 820, in localmax
    x_pad = np.pad(x, paddings, mode='edge')
  File "/usr/lib/python2.7/dist-packages/numpy/lib/arraypad.py", line 1364, in pad
    newmat = _prepend_edge(newmat, pad_before, axis)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/arraypad.py", line 175, in _prepend_edge
    axis=axis)
MemoryError

Answer 1

The Jetson TK1 is a 32-bit processor. It doesn't have sufficient virtual address space to access more than 4GB of RAM from one process.

The kernel can leverage your 16GB page file to provide 4GB of ram to many separate processes, but this still does not expose more than 4GB of addresses to a single process. It simply allows separate processes to individually use up to 4GB of RAM (on Linux, you'll most likely have a 2GB or 3GB limit depending on your kernel settings).

You should split your work into smaller pieces or use a platform with more address space available.

Answer 2

I believe that's because your processor is 32-bit:

The board has the following devices on-board: NVIDIA Tegra124 (Tegra K1 32-bit)

On 32-bit installs of Python, it only has 2 gig of RAM available (as with any 32 bit application by default). Try to re-factor your code accordingly.

No amount of swap space will help this, and relying on swap for large calculations is a really bad idea since it takes a long time. Swap is meant for accidental overflows, and not to be relied on.