在没有for循环的情况下对多维数组的时间序列进行趋势除

Question

I have a 3D array which has a time-series of air-sea carbon flux for each grid point on the earth's surface (model output). I want to remove the trend (linear) in the time series. I came across this code:

from matplotlib import mlab

for x in xrange(40):
    for y in xrange(182):
        cflux_detrended[:, x, y] = mlab.detrend_linear(cflux[:, x, y])

Can I speed this up by not using for loops?

Answer 1

Scipy has a lot of signal processing tools. Using scipy.signal.detrend() will remove the linear trend along an axis of the data. From the documentation it looks like the linear trend of the complete data set will be subtracted from the time-series at each grid point.

import scipy.signal
cflux_detrended = scipy.signal.detrend(cflux, axis=0)

Using scipy.signal will get the same result as using the method in the original post. Using Josef's detrend_separate() function will also return the same result.

Answer 2

Here are two versions using numpy.linalg.lstsq. This version uses np.vander to create any polynomial trend.

Warning: not tested except on the example.

I think something like this will be added to scikits.statsmodels, which doesn't have yet a multivariate version for detrending either. For the common trend case, we could use scikits.statsmodels OLS and we would also get all the result statistics for the estimation.

# -*- coding: utf-8 -*-
"""Detrending multivariate array

Created on Fri Dec 02 15:08:42 2011

Author: Josef Perktold

http://stackoverflow.com/questions/8355197/detrending-a-time-series-of-a-multi-dimensional-array-without-the-for-loops

I should also add the multivariate version to statsmodels

"""

import numpy as np

import matplotlib.pyplot as plt


def detrend_common(y, order=1):
    '''detrend multivariate series by common trend

    Paramters
    ---------
    y : ndarray
       data, can be 1d or nd. if ndim is greater then 1, then observations
       are along zero axis
    order : int
       degree of polynomial trend, 1 is linear, 0 is constant

    Returns
    -------
    y_detrended : ndarray
       detrended data in same shape as original 

    '''
    nobs = y.shape[0]
    shape = y.shape
    y_ = y.ravel()
    nobs_ = len(y_)
    t = np.repeat(np.arange(nobs), nobs_ /float(nobs))
    exog = np.vander(t, order+1)
    params = np.linalg.lstsq(exog, y_)[0]
    fittedvalues = np.dot(exog, params)
    resid = (y_ - fittedvalues).reshape(*shape)
    return resid, params

def detrend_separate(y, order=1):
    '''detrend multivariate series by series specific trends

    Paramters
    ---------
    y : ndarray
       data, can be 1d or nd. if ndim is greater then 1, then observations
       are along zero axis
    order : int
       degree of polynomial trend, 1 is linear, 0 is constant

    Returns
    -------
    y_detrended : ndarray
       detrended data in same shape as original 

    '''
    nobs = y.shape[0]
    shape = y.shape
    y_ = y.reshape(nobs, -1)
    kvars_ = len(y_)
    t = np.arange(nobs)
    exog = np.vander(t, order+1)
    params = np.linalg.lstsq(exog, y_)[0]
    fittedvalues = np.dot(exog, params)
    resid = (y_ - fittedvalues).reshape(*shape)
    return resid, params

nobs = 30
sige = 0.1
y0 = 0.5 * np.random.randn(nobs,4,3)
t = np.arange(nobs)
y_observed = y0 + t[:,None,None]

for detrend_func, name in zip([detrend_common, detrend_separate], 
                               ['common', 'separate']):
    y_detrended, params = detrend_func(y_observed, order=1)
    print '\n\n', name 
    print 'params for detrending'
    print params
    print 'std of detrended', y_detrended.std()  #should be roughly sig=0.5 (var of y0)
    print 'maxabs', np.max(np.abs(y_detrended - y0))

    print 'observed'
    print y_observed[-1]
    print 'detrended'
    print y_detrended[-1]
    print 'original "true"'
    print y0[-1]

    plt.figure()
    for i in range(4):
        for j in range(3):
            plt.plot(y0[:,i,j], 'bo', alpha=0.75)
            plt.plot(y_detrended[:,i,j], 'ro', alpha=0.75)
    plt.title(name + ' detrending: blue - original, red - detrended')


plt.show()

Since Nicholas pointed out scipy.signal.detrend. My detrend separate is basically the same as scipy.signal.detrend with fewer (no axis or breaks) or different (with polynomial order) options.

>>> res = signal.detrend(y_observed, axis=0)
>>> (res - y0).var()
0.016931858083279336
>>> (y_detrended - y0).var()
0.01693185808327945
>>> (res - y_detrended).var()
8.402584948582852e-30

Answer 3

我认为对旧列表的理解最简单：

cflux_detrended = np.array([[mlab.detrend_linear(t) for t in kk] for kk in cflux.T])