AI - Learn the best combination

Question

Total param: 48K

Input X:

array([[ 1964,    12, 32772, ...,     0,  6176,     0],
       [ 1964,    12, 32772, ...,     0,  6841,     0],
       [ 1964,    28, 32772, ...,     0,  6176,     0],
       ...,
       [ 7400,    20, 41565, ...,     0,  8149,     0],
       [ 7400,    20, 41565, ...,     0,  8151,     0],
       [ 7400,    20, 41565, ...,     0,  8150,     0]], dtype=int32)

Output y:

array([0., 0., 0., ..., 1., 0., 0.])

Model structure:

model = Sequential()
model.add(BatchNormalization(input_shape=(7,)))
model.add(Dense(32, activation="relu"))
model.add(Dense(32, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(32, activation="relu"))
model.add(Dense(32, activation="relu"))
model.add(Dense(16, activation="relu"))
model.add(Dense(1, activation=None))

In first input layer i use batch normalization and my acc increase from 50 to 73, which i mean is good solution..

Model compile

model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

i also try 'adam' and same result.

Model fit:

history = model.fit(x_train, y_train, batch_size=2048, validation_data=(x_test, y_test), epochs=1000)

I alo try more combination, with epochs=30000 and batch size 1024 and get acc 78.51.

I also try to double every layer, node*2:

I have 16k with output 1, so with this this solution i get 6k corrected prediction. @ with epochs=30000 and batch size 1024, optimizer adam @

Model structure:

model = Sequential()
model.add(BatchNormalization(input_shape=(7,)))
model.add(Dense(64, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(128, activation="relu"))
model.add(Dense(32, activation="relu"))
model.add(Dense(1, activation='sigmoid'))

My simple question how to increase acc to get more corrected prediction?

Answer 1

If orange is your validation loss / accuracy, you're overfitting. The accuracy almost does not decrease with the number of epochs, while the validation loss increases. Think about

• balancing the labels, maybe they're imbalanced your accuracy doesn't tell you that much
• add early stopping
• adapt the batch size
• adapt the activation function
• add DropOut
• test other optimizers eg Adam

You used a lot of number of hidden layers, reduce them in the beginning. I prefer to start with a small.network, even like logistic regression or a simple linear model, and then have a look if a neural.network increases the performance.

Think about using different methods than neural.networks, for example CART methods (eg XGBOOST) have been shown to outperform neural.networks on problems with small feature size (eg here 7).

I hope that helps to further explore the problem!