operands could not be broadcast together with shapes error in MinMaxScaler

Question

I am developing a prediction model in Python, based on a historical data of previous 45 quarters starting from q-1 of 2010. I am using LSTM for prediction. While trying to implement the following line:

y_perd_future = scaler.inverse_transform(forecast_copies)[:,0]
X = X.copy()
    936             if self.with_std: 
    937                 X *= self.scale_
    938             if self.with_mean:
    939                 X += self.mean_

 ValueError: operands could not be broadcast together with shapes (31,630,3) (32,) (31,630,3) 

Please dont give me examples of small self created arrays. I am looking for advice in context of large data.

Humble Regards

Answer 1

I have found easiest solution, instantiate two scaler one for target variable and other independent variable, simple.

feature = X_train.columns<br>
xscaler = MinMaxScaler()<br>
X_train[feature] = xscaler.fit_transform(X_train[feature])<br>
X_val[feature] = xscaler.transform(X_val[feature])<br>
X_test[feature] = xscaler.transform(X_test[feature])<br>

yscaler = MinMaxScaler()<br>
y_train = yscaler.fit_transform(y_train.values.reshape(-1,1))<br>
y_val = yscaler.transform(y_val.values.reshape(-1,1))

Answer 2

I think the problem is that one tensor has 31 elements (31, 630, 3) while the other 32 (32,) and thus it cannot be broadcasted. At least one dimension must match.

Answer 3

I think the problem is what is scaled first, First of all always scale, target and features separately I always scale independent variables first & then scale target variable later, I don't know why but MinMaxscaler or any scaling technique takes in to account what is scaled last. After prediction you can use inverse_transform fro ex. pressure = scaler.inverse_transform(y_train_pred_lr.values.reshape(-1,1)) print(pressure) Better to scale target & features separately,scale target after independent variable Hope this solves your problem, let me know if it does.

If you scale features first & scale target later, this will give you broadcast error

Answer 4

Better to use two different instance of scaling for example, 'scaler' for the independent variables and 'scaler1' for dependent ie target variable.