量化神经网络python的预测质量

Question

Today i'd like to ask you some help with the quality of my neural network. I've been working with a project to predict parameters in metallurgy. To make sure that my neural network is going on the right way i tried to use some functions of "Scikit-learn" like "score" and "r^2" but no success.

With the actual code my "r²" is -10.42239374572942, this value is unreal because everybody knows the r² must be between -1 and 1.

Anyone have any suggestion to evaluate my neural network? Why my code is not working?

Thaks guys. See you.

Follow above my code:

# coding: utf-8

import pandas as pd
import numpy as np

#modulo de plot
import matplotlib.pyplot as plt

#modulo da rede propriamente dita
from sklearn.neural_network import MLPRegressor

#para testar a rede neural
from sklearn.model_selection import train_test_split

#para normalização
from sklearn.preprocessing import StandardScaler

#para testar a qualidade da rede neural
from sklearn.metrics import mean_squared_error, r2_score

#buscando o CSV com os dados do AF1-Gerdau
df = pd.read_csv('Rede3.03.11.17_MOACIR_b.csv', delimiter=';', encoding = "ISO-8859-1" )

df2 = df.dropna(how='all')


# ## Definindo as variáveis inputs e a resposta

X = df2.drop(['Fuel Rate'], axis=1) #deixando todas as colunas exceto a variável resposta "Fuel Rate"
y = df2['Fuel Rate'] #variável respota "Fuel Rare"

# ## Normalizando os dados para uma melhor convergência

scaler = StandardScaler()
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Treinamento apenas com os dados de treino
scaler.fit(X_train)

# Aplicando a transformação de normalização dos dados:
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

# ## Criando os parametros da RNA
rna = MLPRegressor(hidden_layer_sizes=(13,13,13), max_iter=2000)

# ## Treinando a RNA
rna.fit(X_train,y_train)

# ## Testando a rede
y_predicted = rna.predict(X_test)

# The coefficients
print('Coefficients: \n', r2_score(y_test, y_predicted))

Answer 1

To start with, the r2_score() metric can be arbitrarily low. It need not be between -1 and 1 and does not correspond to the pearson correlation coefficient. A score of ~-10 then just means that the model is performing significantly worse than a model which outputs mean values (corresponding to an r2 score of 0).

The r2_score() you chose is a fine preliminary metric for your model. The output looks strange not because there is a problem with the metric but because there is a problem with the model. In its current state, improving r2_score() will probably improve any other metric of interest. You might also be interested in the similarity between metrics on your test set and metrics on your train set -- the more similar they are the more likely your model is to generalize well (under the manifold hypothesis or any other framework avoiding no free lunch theorems). Depending on the intended application, you might care about the worst case scenario -- it might not matter if you have an r2 of 0.999 if the worst case is that a self driving car mistakes a pedestrian for a safe place to drive. You might be interested in a wide range of calibration scores -- if your model predicts a strong pattern with good accuracy but the residuals demonstrate a strong bias in the model, something may very well be amiss. In general, the quality of a model depends strongly on its intended application, and your metrics should reflect your end goals. Using The Case for Learned Index Structures as an example, sometimes generalization is a bad thing and not what you want at all.

Several potential problems exist in your approach. I'm not quite sure about metallurgy, but many chemistry problems have chaotic interaction dynamics and are not easily modeled with a naive approach which just slaps a model onto the input data (naive is not meant negatively here -- the naive approach is often a good starting point and may completely satisfy the requirements at hand).

As a rule of thumb, most of the gains you will find are going to be in feature engineering. In the case of chemical problems, this can involve taking the outputs from standard solutions to the problem (which are obviously insufficient since you're going straight for machine learning) and treating them as features in your neural network. The idea is that even though no individual model is perfect, they're all right often enough and wrong in different ways so that the neural network is able to figure out how to combine them together to create a better answer. This is a type of ensemble technique.

What data visualizations/analysis have you done? Do you have an idea of which features correspond to the output you're trying to predict? It's even possible that the input you have does not have enough information to predict your desired output. Have you examined the data before dropping the NaN values? Would a better imputation method yield additional gains? Your model is coded fine. Understanding your data is easily the most important task at hand.

Scikit-learn can have problems optimizing networks with small numbers of nodes in each layer. A grid search and cross validation procedure to help determine the optimal parameters may improve your model substantially. The solution might be sensitive to tolerances and other such parameters.