Keras Error with fit_generator: Error when checking target: expected softmax_1 to have shape (2,) but got array with shape (1,)

Question

I am new to using keras. My dataset is large so I am reworking my code to work in batches. I have my generator here:

def batch_generator(csv_file,chunk_size,
                    steps, var_list):
    idx=1
    while True:
        yield load_data(csv_file,idx-1,chunk_size,var_list)## Yields data
        if idx<steps:
            idx+=1
        else:
            idx=1

def load_data(csv_file,idx,
              chunk_size, var_list):
    global col_names
    if idx == 0:
        df = pd.read_csv(
                  csv_file,
                  nrows=chunk_size)
        col_names = df.columns
    else:
    df = pd.read_csv(
                  csv_file, skiprows=idx*chunk_size,
                  nrows=chunk_size,
                  header=None,names = col_names)
    x = df[var_list]
    y = df['targets_LJ']
    return (np.array(x), to_categorical(y))

And the machine learning part of my code:

    #create iterator over dataframe
    train_gen = batch_generator(filepath_train, chunk_size, steps, list_of_vars)
    val_gen = batch_generator(filepath_val, chunk_size, steps_val, list_of_vars)

    # now make the network
    from keras.layers import Input, Dense, Softmax
    from keras.models import Model

    #layers are functions that construct the deep learning model
    #tensors define the data flow through the model
    input_tensor = Input(shape = (len(list_of_vars),))
    node1_layer = Dense(2)
    node1_tensor = node1_layer(input_tensor)
    output_layer = Softmax()
    output_tensor = output_layer(node1_tensor)

    #build model
    model = Model(input_tensor, output_tensor)
    model.compile(optimizer='rmsprop',
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])

    #create early stopping for if the nn is not improving
    from keras.callbacks import EarlyStopping
    early_stop = EarlyStopping(monitor='val_loss', patience=2)

    #fit model
    history = model.fit_generator(generator=train_gen,
        validation_data=val_gen,
        steps_per_epoch=steps, epochs=args.epochs, validation_steps=steps_val, callbacks=[early_stop])

I am getting an error that I was not getting before I switched from fit to fit_generator:

Traceback (most recent call last):
  File "./train_nn.py", line 162, in <module>
    run()
  File "./train_nn.py", line 145, in run
    steps_per_epoch=steps, epochs=args.epochs, validation_steps=steps_val, callbacks=[early_stop])
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/engine/training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1211, in train_on_batch
    class_weight=class_weight)
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 789, in _standardize_user_data
    exception_prefix='target')
  File "/opt/ohpc/pub/packages/anaconda3/lib/python3.7/site-packages/keras/engine/training_utils.py", line 138, in standardize_input_data
    str(data_shape))
ValueError: Error when checking target: expected softmax_1 to have shape (2,) but got array with shape (1,)

I am lost as to what is wrong with this. I am using 'categorical_crossentropy' but I have my targets as categorical and as far as I know these should work together.

Thanks, Sarah

Answer 1

Your model has output shape (2,) , because your last layer has 2 units.
Since you're using "softmax" , I presume you're doing a binary classification, right?

But your data has shape (1,) , meaning you don't have two classes!!! You have only one class. In usual binary classifications, you got data as zeros (one class) and ones (another class)

If this is your case, your last layer must contain 1 unit only. Your last activation should be 'sigmoid' , and your loss should be 'binary_crossentropy' . This way you don't need to change your data.