过滤数据时出现MemoryError

Question

I am implementing a filtfilt filter, but I am getting a MemoryError. I have tried with two different computers and the error remains.

  File "C:\Users\UserName\lib\site-packages\scipy\signal\signaltools.py", line 3129, in filtfilt
    zi = lfilter_zi(b, a)
  File "C:\Users\UserName\lib\site-packages\scipy\signal\signaltools.py", line 2689, in lfilter_zi
    IminusA = np.eye(n - 1) - linalg.companion(a).T
  File "C:\Users\UserName\lib\site-packages\numpy\lib\twodim_base.py", line 201, in eye
    m = zeros((N, M), dtype=dtype, order=order)
MemoryError

Any clues how to solve this? My data isn't that huge (1D array with length 3011723). I'm using Pycharm at python win63. I'd be very grateful for help.

This is a sample code where the error occurs:

from numpy import arange, random
from scipy.signal import kaiserord, firwin, filtfilt

fs = 1000
data = random.uniform(size=3011723)

nyq_rate = fs / 2.0

# The desired width of the transition from pass to stop, relative to the Nyquist rate
width = 0.1/nyq_rate

# The desired attenuation in the stop band, in dB.
ripple_db = 100.0

# Compute the order and Kaiser parameter for the FIR filter.
N, beta = kaiserord(ripple_db, width)

# Use firwin with a Kaiser window to create a lowpass FIR filter.
taps = firwin(N, 300/nyq_rate, window=('kaiser', beta), pass_zero=True)

# Use lfilter to filter x with the FIR filter.
filtered_data = filtfilt(taps, 1.0, data)

Best, DTake

Answer 1

Your problem occurs when scipy.signal.filtfilt tries to compute the companion matrix. For the intermediate calculation, it creates an array of size (3011723, 3011723) , which is roughly around 72563 GB . It is clear that it can't fit into the memory.

I don't see any other option but to reduce the sample size.