为什么重塑我的数据会完全改变 Keras 中全连接神经网络的行为？

Question

I would love some insight on this. I'm working on a regression problem in Keras with a simple neural network. I have train and test data, training data consists of 33230 samples with 20020 features (which is a ton of features for this amount of data, but that's another story - the features are just various measurements). Test set is 8308 samples with same number of features. My data is in a pandas dataframe, and I convert it into numpy arrays which seem to look as expected:

X_train = np.array(X_train_df)
X_train.shape

(33230, 20020)

X_test = np.array(X_test_df)
X_test.shape

(8308, 20020)

If I pass this into the following fully connected model, it trains very quickly, and produces terrible results on the test set:

Model:

model = Sequential()
model.add(Dense(300, activation="relu", input_shape=(20020,)))
model.add(Dense(300, activation="relu"))
model.add(Dense(100, activation="relu"))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss='mse',  metrics=['mean_absolute_error'])

Fit:

model.fit(x=X_train, y=y_train, validation_data=(X_test,  y_test), batch_size=128, shuffle=True, epochs=100)

Results after 5 epochs (doesn't change substantially after this, training loss goes down, validation loss shoots up):

Train on 33230 samples, validate on 8308 samples
Epoch 1/100
33230/33230 [==============================] - 11s 322us/sample - loss: 217.6460 - mean_absolute_error: 9.6896 - val_loss: 92.2517 - val_mean_absolute_error: 7.6400
Epoch 2/100
33230/33230 [==============================] - 10s 308us/sample - loss: 70.0501 - mean_absolute_error: 7.0170 - val_loss: 90.1813 - val_mean_absolute_error: 7.5721
Epoch 3/100
33230/33230 [==============================] - 10s 309us/sample - loss: 62.5253 - mean_absolute_error: 6.6401 - val_loss: 104.1333 - val_mean_absolute_error: 8.0131
Epoch 4/100
33230/33230 [==============================] - 11s 335us/sample - loss: 55.6250 - mean_absolute_error: 6.2346 - val_loss: 142.8665 - val_mean_absolute_error: 9.3112
Epoch 5/100
33230/33230 [==============================] - 10s 311us/sample - loss: 51.7378 - mean_absolute_error: 5.9570 - val_loss: 208.8995 - val_mean_absolute_error: 11.4158

However if I reshape the data:

X_test = X_test.reshape(8308, 20020, 1)
X_train = X_train.reshape(33230, 20020, 1)

And then use the same model with a Flatten() after the first layer:

model = Sequential()
model.add(Dense(300, activation="relu", input_shape=(20020,1)))
model.add(Flatten())
model.add(Dense(300, activation="relu"))
model.add(Dense(100, activation="relu"))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss='mse',  metrics=['mean_absolute_error'])

Then my results look very different, and much better:

Train on 33230 samples, validate on 8308 samples
Epoch 1/100
33230/33230 [==============================] - 1117s 34ms/sample - loss: 112.4860 - mean_absolute_error: 7.5939 - val_loss: 59.3871 - val_mean_absolute_error: 6.2453
Epoch 2/100
33230/33230 [==============================] - 1112s 33ms/sample - loss: 4.7877 - mean_absolute_error: 1.6323 - val_loss: 23.8041 - val_mean_absolute_error: 3.8226
Epoch 3/100
33230/33230 [==============================] - 1116s 34ms/sample - loss: 2.3945 - mean_absolute_error: 1.1755 - val_loss: 14.9597 - val_mean_absolute_error: 2.8702
Epoch 4/100
33230/33230 [==============================] - 1113s 33ms/sample - loss: 1.5722 - mean_absolute_error: 0.9616 - val_loss: 15.0566 - val_mean_absolute_error: 2.9075
Epoch 5/100
33230/33230 [==============================] - 1117s 34ms/sample - loss: 1.4161 - mean_absolute_error: 0.9179 - val_loss: 11.5235 - val_mean_absolute_error: 2.4781

It also takes 1000x times longer, but performs well on the test set. I don't understand why this happens. Can someone shed light on this? I'm guessing I'm missing something really basic, but I can't figure out what.

Answer 1

A very good question. First of all you will have to understand how the network actually work. Dense layer is a fully conected layer so each neuron will have a connection with the previous layer's neuron. Now your networks Performance that you have mentioned that it is 1000x time slower is nothing to do with your training data, but with your network. Your second network is so big that I was unable to fit it in my RAM as well as not in Google Colab. So for demonstration purposes I will take that your training data is is of (500, 100) shape.

For the First network that you posted taking the above mentioned shape your model network looks something like below:

model = Sequential()
model.add(Dense(300, activation="relu", input_shape=(100,)))
model.add(Dense(300, activation="relu"))
model.add(Dense(100, activation="relu"))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss='mse',  metrics=['mean_absolute_error'])


Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_2 (Dense)              (None, 300)               30300     
_________________________________________________________________
dense_3 (Dense)              (None, 300)               90300     
_________________________________________________________________
dense_4 (Dense)              (None, 100)               30100     
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 101       
=================================================================
Total params: 150,801
Trainable params: 150,801
Non-trainable params: 0
_________________________________________________________________

Take a note of the Total params, it is 150,801 . Now if we take your second example.

model1 = Sequential()
model1.add(Dense(300, activation="relu", input_shape=(100,1)))
model1.add(Flatten())
model1.add(Dense(300, activation="relu"))
model1.add(Dense(100, activation="relu"))
model1.add(Dense(1, activation='linear'))
model1.compile(optimizer='adam', loss='mse',  metrics=['mean_absolute_error'])

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_14 (Dense)             (None, 100, 300)          600       
_________________________________________________________________
flatten_2 (Flatten)          (None, 30000)             0         
_________________________________________________________________
dense_15 (Dense)             (None, 300)               9000300   
_________________________________________________________________
dense_16 (Dense)             (None, 100)               30100     
_________________________________________________________________
dense_17 (Dense)             (None, 1)                 101       
=================================================================
Total params: 9,031,101
Trainable params: 9,031,101
Non-trainable params: 0
_________________________________________________________________

Your total params increases to 9,031,101 . You can image when you use your actual data that has length 20020 . Your model increases like anything and I was even unable to fit that model in my RAM.

So to conclude, your second model has huge number of parameters compared to first model. This is the reason for slow training and better performance may be? more parameters makes the learning better. Can't say what makes it better without actually looking at your data. But more parameters can contribute to better performance.

Note: If you remove the Flatten layer your network paramters will decrease, here is the example.

model1 = Sequential()
model1.add(Dense(300, activation="relu", input_shape=(100,1)))
model1.add(Dense(300, activation="relu"))
model1.add(Dense(100, activation="relu"))
model1.add(Dense(1, activation='linear'))
model1.compile(optimizer='adam', loss='mse',  metrics=['mean_absolute_error'])

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_18 (Dense)             (None, 100, 300)          600       
_________________________________________________________________
dense_19 (Dense)             (None, 100, 300)          90300     
_________________________________________________________________
dense_20 (Dense)             (None, 100, 100)          30100     
_________________________________________________________________
dense_21 (Dense)             (None, 100, 1)            101       
=================================================================
Total params: 121,101
Trainable params: 121,101
Non-trainable params: 0
_________________________________________________________________

I hope my answer helped you understand what is hapening and what is the difference between two models.

UPDATE: 20/07 For your comment, I thought it is better to update the answer for more clarity. Your question is -- how does the number of parameters relate to the shape of the network?

To be honest I do not clearly understand what you mean by this. I will still try to answer it. The more layers or neurons you add increases the network and the number of trainable parameters.

So your actual issue is why does the layer Flatten increases you parameter. For that you need to understand how are parameters calculated.

model.add(Dense(300, activation="relu", input_shape=(100,)))

Consider this is your first layer the number of parameters will be units *(input_size + 1) that comes to 30300 . Now when you add the Flatten layer, this actually does not increase your parameter by itself, but the output of the Flatten layer is input to the following layer. So consider the following example.

_________________________________________________________________
flatten_2 (Flatten)          (None, 30000)             0         
_________________________________________________________________
dense_15 (Dense)             (None, 300)               9000300   
_________________________________________________________________

Here you can see that the output size of the Flatten layer is 30000 . Now considering the above formula you can see 300 *(30000 + 1) will result in 9000300 parameters which is a huge deal in itself. More number of parameters can help to learn more features and might help to achieve better results. But it always depends on the data, you will have to experiment with it.

I hope the above explainations might have cleared your doubts.