Keras Assertion Error while Fitting Model

Question

I am trying to replicate this model on my dataset: https://docs.seldon.io/projects/alibi/en/stable/examples/cem_iris.html

TF.Version: 2.8.2 Keras Vesion: 2.8.0 I'm getting "Assertion Error" when I try to fit the model. My dataset has 59columns and the target variable has 3-classes.

The code:

df=pd.read_csv('file.csv')
df= df.dropna(subset=['column names'])

X = df.drop(columns=['target'], axis = 1)
y = df['target']

num_pipeline = Pipeline([
        ('std_scaler', StandardScaler())             
    ])

X = num_pipeline.fit_transform(X)
idx = 1000
x_train,y_train = X[:idx,:], y[:idx]
x_test, y_test = X[idx+1:,:], y[idx+1:]
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

def lr_model():
    x_in = Input(shape=(59,))
    x_out = Dense(3, activation='softmax')(x_in)
    lr = Model(inputs=x_in, outputs=x_out)
    lr.compile(loss='categorical_crossentropy',
               optimizer='rmsprop', metrics=['accuracy'])
    return lr

lr = lr_model()
lr.summary()
lr.fit(x_train, y_train, batch_size=180, epochs=500, verbose=0)

The output and the full error traceback:

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_37 (InputLayer)       [(None, 59)]              0         
                                                                 
 dense_36 (Dense)            (None, 3)                 180       
                                                                 
=================================================================
Total params: 180
Trainable params: 180
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-123-0216109227fb> in <module>
     15 lr = lr_model()
     16 lr.summary()
---> 17 lr.fit(x_train, y_train, batch_size=181, epochs=500, verbose=0)

7 frames
/usr/local/lib/python3.7/dist-packages/keras/engine/training_v1.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    794         max_queue_size=max_queue_size,
    795         workers=workers,
--> 796         use_multiprocessing=use_multiprocessing)
    797 
    798   def evaluate(self,

/usr/local/lib/python3.7/dist-packages/keras/engine/training_generator_v1.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    775         shuffle=shuffle,
    776         initial_epoch=initial_epoch,
--> 777         steps_name='steps_per_epoch')
    778 
    779   def evaluate(self,

/usr/local/lib/python3.7/dist-packages/keras/engine/training_generator_v1.py in model_iteration(model, data, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch, mode, batch_size, steps_name, **kwargs)
    250 
    251       is_deferred = not model._is_compiled
--> 252       batch_outs = batch_function(*batch_data)
    253       if not isinstance(batch_outs, list):
    254         batch_outs = [batch_outs]

/usr/local/lib/python3.7/dist-packages/keras/engine/training_v1.py in train_on_batch(self, x, y, sample_weight, class_weight, reset_metrics)
   1061           y,
   1062           sample_weights=sample_weights,
-> 1063           output_loss_metrics=self._output_loss_metrics)
   1064       outputs = (output_dict['total_loss'] + output_dict['output_losses']
   1065                  + output_dict['metrics'])

/usr/local/lib/python3.7/dist-packages/keras/engine/training_eager_v1.py in train_on_batch(model, inputs, targets, sample_weights, output_loss_metrics)
    310           sample_weights=sample_weights,
    311           training=True,
--> 312           output_loss_metrics=output_loss_metrics))
    313   if not isinstance(outs, list):
    314     outs = [outs]

/usr/local/lib/python3.7/dist-packages/keras/engine/training_eager_v1.py in _process_single_batch(model, inputs, targets, output_loss_metrics, sample_weights, training)
    245       ValueError: If the model has no loss to optimize.
    246   """
--> 247   with backend.eager_learning_phase_scope(1 if training else 0), \
    248       training_utils.RespectCompiledTrainableState(model):
    249     with GradientTape() as tape:

/usr/lib/python3.7/contextlib.py in __enter__(self)
    110         del self.args, self.kwds, self.func
    111         try:
--> 112             return next(self.gen)
    113         except StopIteration:
    114             raise RuntimeError("generator didn't yield") from None

/usr/local/lib/python3.7/dist-packages/keras/backend.py in eager_learning_phase_scope(value)
    590   global _GRAPH_LEARNING_PHASES  # pylint: disable=global-variable-not-assigned
    591   assert value in {0, 1}
--> 592   assert tf.compat.v1.executing_eagerly_outside_functions()
    593   global_learning_phase_was_set = global_learning_phase_is_set()
    594   if global_learning_phase_was_set:

Answer 1

The way you import data as dataframe and If you are right that the dataset contains 59 features as you mentioned above; i will think it only requires to change the code here, where you slice/split dataset for training and testing;

x_train,y_train = X[:idx,:], y[:idx]
x_test, y_test = X[idx+1:,:], y[idx+1:]

and should be changed into

x_train,y_train = X[:idx], y[:idx]
x_test, y_test = X[idx+1:], y[idx+1:]

Or

x_train,y_train = X.iloc[:idx,:], y[:idx]
x_test, y_test = X.iloc[idx+1:,:], y[idx+1:]

But notice, as you know, you are trying to scale your dataset in the way that occur data leakage, cause of using pipline for whole dataset; While you should fit_transform for training set and then transform for testing set.