使用Scikit学习管道，当要素依赖于其他行时，如何从时间序列数据中生成要素？

Question

Here is a concrete example scenario:

I want to train a classifier to predict whether a given stock will go up or down in price the next day.

Here is how I want to do that:

I have the daily close price of a particular stock over a given period of time. I want to generate features including the exponential weighted moving average and price rate of change . These calculations for a given day require the close prices of previous days. Then I want to calculate a target variable 1 or -1 indicating if the stock price went up or down the next day.

After generating the features and target, I want to split the data into train/test (or even train/validation/test) groups then train and test a classifier to predict the target.

Finally, I want to implement and execute these steps in an sklearn Pipeline for two main reasons: 1.) to easily manipulate the data flow and/or try different classifiers and 2.) to run a grid search to find good parameters to use in both the feature generation steps and classifier -- eg how many days should be taken into account to calculate the exponential weighted moving average or how many estimators should be used in the random forest classifier?

Here is the issue I run into:

From what I've read about sklearn Pipelines, I would need to create custom transformations and perhaps use FeatureUnion to generate the features. However, the examples I've seen all call .fit(X_train, y_train) , which runs each step in the Pipeline (including generating the features). But my features depend on other rows which may not be in X_train.

Answer 1

Have a look at the documentation for tsfresh . They even have an example for what you are looking for: using time series transforms with sklearn pipelines'