How can i make GradientBoostingRegressor work with a BaseEstimator in scikit-learn?

Question

Sklearn for gbm supports init parameter which gives an option to train an initial model and pass it within another model using the init param.

I am trying to use the same concept for regression. Below is my code.

gbm_base=GradientBoostingRegressor(random_state=1,verbose=True)
gbm_base.fit(X_train, y_train)
gbm_withEstimator= 
GradientBoostingRegressor(init=gbm_base,random_state=1,verbose=True)
gbm_withEstimator.fit(X_train, y_train)

But it is giving me the following error.

~/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/gradient_boosting.py in
update_terminal_regions(self, tree, X, y, residual, y_pred, sample_weight, sample_mask, learning_rate, k)

499         """
500         # update predictions

--> 501         y_pred[:, k] += learning_rate * tree.predict(X).ravel()

502 
503     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,

IndexError: too many indices for array

I think it is getting the error because in regression ypred is always a one dimensional array but in the code here it is assuming it to be a two dimensional

Answer 1

This is a known bug. Take a look at GradientBoosting fails when using init estimator parameter. and [MRG] FIX gradient boosting with sklearn estimator as init #12436 for more context.

In the meantime you can subclass GradientBoostingRegressor to avoid the issue as follows:

from sklearn.utils import check_array


class GBR_Init(GradientBoostingRegressor):
    def predict(self,X):
        X = check_array(X, dtype=np.float32, order='C', accept_sparse='csr')
        return self._decision_function(X)

Then you can use the GBR_Init class instead of the GradientBoostingRegressor.

An example:

import numpy as np
from sklearn.datasets import load_boston
from sklearn.ensemble import GradientBoostingRegressor as GBR
from sklearn.utils import check array

class GBR_Init(GradientBoostingRegressor):
    def predict(self,X):
        X = check_array(X, dtype=np.float32, order='C', accept_sparse='csr')
        return self._decision_function(X)

boston = load_boston()
X = boston.data
y = boston.target
base = GBR_Init(random_state=1, verbose=True)
base.fit(X, y)
      Iter       Train Loss   Remaining Time
         1          71.3024            0.00s
         2          60.6243            0.00s
         3          51.6694            0.00s
         4          44.3657            0.00s
         5          38.2831            0.00s
         6          33.2863            0.00s
         7          28.9190            0.00s
         8          25.2967            0.18s
         9          22.2587            0.16s
        10          19.6923            0.14s
        20           8.3119            0.13s
        30           5.4763            0.07s
        40           4.1906            0.07s
        50           3.4663            0.05s
        60           3.0437            0.04s
        70           2.6753            0.03s
        80           2.4451            0.02s
        90           2.2376            0.01s
       100           2.0142            0.00s
GBR_Init(alpha=0.9, criterion='friedman_mse', init=None, learning_rate=0.1,
     loss='ls', max_depth=3, max_features=None, max_leaf_nodes=None,
     min_impurity_decrease=0.0, min_impurity_split=None,
     min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0,
     n_estimators=100, n_iter_no_change=None, presort='auto',
     random_state=1, subsample=1.0, tol=0.0001, validation_fraction=0.1,
     verbose=True, warm_start=False)
est = GBR_Init(init=base, random_state=1, verbose=True)
est.fit(X, y)
est.fit(X, y)
      Iter       Train Loss   Remaining Time
         1          71.3024            0.00s
         2          60.6243            0.00s
         3          51.6694            0.00s
         4          44.3657            0.00s
         5          38.2831            0.00s
         6          33.2863            0.00s
         7          28.9190            0.00s
         8          25.2967            0.18s
         9          22.2587            0.16s
        10          19.6923            0.14s
        20           8.3119            0.06s
        30           5.4763            0.07s
        40           4.1906            0.05s
        50           3.4663            0.05s
        60           3.0437            0.03s
        70           2.6753            0.03s
        80           2.4451            0.02s
        90           2.2376            0.01s
       100           2.0142            0.00s
      Iter       Train Loss   Remaining Time
         1           2.0069            0.00s
         2           1.9844            0.00s
         3           1.9729            0.00s
         4           1.9670            0.00s
         5           1.9409            0.00s
         6           1.9026            0.00s
         7           1.8850            0.00s
         8           1.8690            0.00s
         9           1.8450            0.00s
        10           1.8391            0.14s
        20           1.6879            0.06s
        30           1.5695            0.04s
        40           1.4469            0.05s
        50           1.3431            0.03s
        60           1.2329            0.03s
        70           1.1370            0.02s
        80           1.0616            0.02s
        90           0.9904            0.01s
       100           0.9228            0.00s
GBR_Init(alpha=0.9, criterion='friedman_mse',
     init=GBR_Init(alpha=0.9, criterion='friedman_mse', init=None, learning_rate
=0.1,
     loss='ls', max_depth=3, max_features=None, max_leaf_nodes=None,
     min_impurity_decrease=0.0, min_impurity_split=None,
     min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0,
     n_estimators=100, n_iter_no_change=None, presort='auto',
     random_state=1, subsample=1.0, tol=0.0001, validation_fraction=0.1,
     verbose=True, warm_start=False),
     learning_rate=0.1, loss='ls', max_depth=3, max_features=None,
     max_leaf_nodes=None, min_impurity_decrease=0.0,
     min_impurity_split=None, min_samples_leaf=1, min_samples_split=2,
     min_weight_fraction_leaf=0.0, n_estimators=100, n_iter_no_change=None,
     presort='auto', random_state=1, subsample=1.0, tol=0.0001,
     validation_fraction=0.1, verbose=True, warm_start=False)