通过预处理提高神经网络的准确性

Question

Reading https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607

states to debug a neural network check following :

5. Is the relationship between input and output too random? Maybe the non-random part of the relationship between the input and output is too small compared to the random part (one could argue that stock prices are like this). Ie the input are not sufficiently related to the output. There isn't an universal way to detect this as it depends on the nature of the data.

To check this I wrote below code :

my dataframe :

columns = ['A','B']
data = np.array([[1,2] , [1,5], [2,3], [2,3]])
df = pd.DataFrame(data,columns=columns)
df
    A   B
0   1   2
1   1   5
2   2   3
3   2   3

Where A is input variable and B is target variable.

Code which measures predictive power for label 1 :

df_sub1 = df[df['A'] == 1] 
len(df_sub1['A'].unique()) / len(df_sub1['B'].unique())

Value returned is 0.5 as for label 1 there are two different target values..

Code which measures predictive power for label 2 :

df_sub1 = df[df['A'] == 2] 
len(df_sub1['A'].unique()) / len(df_sub1['B'].unique())

Value returned is 1 as for label 2 both target values are same.

From this can reason that attribute 1 is a better predictor than attribute 2 ? I created this from reading above "Is the relationship ...." . This calculation has a title and is it a good measure of predictability ?

To improve accuracy of neural network through data pre-processing can try removing values from training set where predictive power is below a pre-defined threshold value, where value is result of above calculations?

Answer 1

I do not understand your quote the same way you did. So let's distinguish both interpretations.

1. According to you, you qualify the random part of your model as a subset of your predictors (A) that leads to random outputs (B), and should be therefore removed.

2. In my opinion, the quote should be interpreted as the general relationship between predictors (A) and target variables (B)

These are two different things.

Interpretation 1

If you remove your set {A=1} from your prediction set, you have to remove it also from your prediction set. Basically, you will train your neural network to predict B only when A is not 1. As the outcome of B is uncertain when A = 1, your model performance is likely to increase but what if you have to cast prediction when cases A = 1 occurs?

Indeed, you have increased accuracy but you have reduced your prediction potential to {A!=1} and the operation is only worth it if you find another model that would beat your neural network when {A=1} so that you general accuracy is higher. Besides, given the neural network non linear structure, it should theoretically capable of making the distinction between the two case by itself, so I have doubts on the pertinence of such approach.

Regarding your attempt of measuring the predictive power, you must be aware that there is no predictive power without predictive method or model. By using the unique method, you make strong assumption on the equiprobabilities of your outputs. How would your predictive power react with the following data?

data = np.array([[1,2] , [1,5], [2,3], [2,3], [2,3], [2,4]])
df1 = pd.DataFrame(data[:-2,:], columns=columns) # your data
df2 = pd.DataFrame(data, columns=columns) # my data

# your method applied to my data
print 1 / df2.groupby('A')['B'].nunique()

Prints

A
1    0.5
2    0.5
Name: B, dtype: float64

Both value of A leads to the same predictive power but in case {A=1} outcomes are equiprobable and with {A=2}, in terms of maximum likelihood, the prediction should be 3.

The main problem is that you have a model in mind to represent the predictive power that is different from the model you intend to use, ie the neural network. So, if you want to measure the predictive power of your variable (generally or with some conditional constraint) why not simply use the model itself?

Otherwise, if you want to use a fast proxy to measure how the value of predictor reduces the uncertainty about a variable you have more robust metrics at your disposal, such as the information gain that is easy to implement and already use in decision trees to split node into branches.

I let your read about it but here is an example to show how it overcomes the above problem:

# information gain method

def entropy(data):
    """Compute entropy of a set of values"""
    bin_cnt  = np.bincount(data)
    bin_prob = bin_cnt / np.sum(bin_cnt)
    entropy = - np.sum(bin_prob * np.ma.log2(bin_prob))
    return entropy

# using your data
print entropy(df1['B']) - df1.groupby('A')['B'].apply(entropy)

prints

A
1    0.5
2    1.5
Name: B, dtype: float64

Showing that we have more information gain when A=2.

# Using my data
print entropy(df2['B']) - df2.groupby('A')['B'].apply(entropy)

prints

A
1    0.792481
2    0.981203
Name: B, dtype: float64

Showing that we have still more information gain when A=2.

Interpretation 2

the input are not sufficiently related to the output.

As I mentioned, I do not believe that it should be regarded as subset of the input-output as you did but in their overall relationship. Assuming a deterministic predicted phenomenon, I see three different cases where the input and the output relationship can be weak generally:

1. Your predictors are weak proxies of the explanatory variables of the predicted phenomenon
2. Your predictors are noisy
3. Your predicted phenomenon is high dimensional (explained by a lot of factor) and maybe nonlinear (ie. even more sensitive too noise, as it is more difficult to explain the process)

You may observe this 3 cases together and what you should do are the usual but challenging tasks of: finding more representative data, decomposing and denoising, reducing dimension, select a model that fit complex behaviors. And indeed, all these tasks ...

depends on the nature of the data