sklearn 中神经网络预测的波动

Question

When I fit a sklearn neural network (MLPRegressor) to a very small dataset(50 elements) and 18 input features. I get a highly fluctuating output each time, I run the neural network. The reason is, that the parameters (weights and biases) are initialized each time and the number of inputs apparently is not sufficient to fit the neural network. I have to stress, that the MLP regressors max iterations is set to 2000 and hence should converge. It is probably a problem of overfitting, but it fluctuates for different layer - node combinations. How to tackle this problem?

---

Edit:

The error is large compared to a random forrest (0.2 eV), to a linear polynomial fit (0.18 eV) and Gaussian regression (0.17). The RMSE is plotted, the target variable is in the interval [-1,2].

The neural network has a small size of 3*2 According to the design principle here , the phenomenon does however not change for other geometries. A hyperparameter search is difficult if the results are fluctuating that much.

Answer 1

As a non-expert in neural networks but with experience in statistical modelling, which seems like it would apply here: when you say it "should converge," why should it converge? And have you checked how many layers and how many weights/neurons per layer you're using? Or how many total independent parameters?

Because based on how neural networks usually work, you're trying to find a maximum in a space with some huge number of dimensions, and many many local maxima. On top of that, you don't have enough data for the model to fit well, possibly fewer data points than independent parameters. I can't imagine that there are any optimization methods that should reliably converge to the same results over multiple runs in such a situation.

In a normal statistical model, the least you would do in this case is use a simpler model, with fewer parameters. In your case, perhaps the question to ask is whether it's really feasible to fit any kind of MLP regressor to 50 data points?

It's hard to answer your question without knowing the form of the data you're trying to fit. But with a small dataset, you could try another type of model for whatever you're doing. Neural networks only see benefits over other types of models for large datasets.