Unable to save the model made with Keras functional API

Question

Specifically trying to save the entire MaskRCNN model implemented here https://github.com/matterport/Mask_RCNN

In line 2343 of https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py basically changed save_weights_only to False from True so that entire model is saved.

keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=False),

Stack trace of error is below

  File "./samples/coco/coco.py", line 509, in <module>
    augmentation=augmentation)
  File "/mask_rcnn_root/Mask_RCNN/mrcnn/model.py", line 2374, in train
    use_multiprocessing=True,
  File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1415, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training_generator.py", line 247, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/usr/local/lib/python3.5/dist-packages/keras/callbacks.py", line 77, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/usr/local/lib/python3.5/dist-packages/keras/callbacks.py", line 455, in on_epoch_end
    self.model.save(filepath, overwrite=True)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 1085, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/saving.py", line 116, in save_model
    'config': model.get_config()
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 926, in get_config
    return copy.deepcopy(config)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 218, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 223, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.5/copy.py", line 223, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 223, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.5/copy.py", line 223, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.5/copy.py", line 182, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python3.5/copy.py", line 297, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.5/copy.py", line 182, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python3.5/copy.py", line 297, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.5/copy.py", line 174, in deepcopy
    rv = reductor(4)
TypeError: can't pickle SwigPyObject objects keras 

Thank you!

Answer 1

Basically the reason is improper use of Lambda layers in Keras messes up the saving of the model. Weights can still be saved though using model.save_weights("my_model.h5") but if you try to save the entire model or extract the graph structure then the you get the crash. So all of the below failed for my case

model.save('my_model.h5') 
json_string = model.to_json() 
yaml_string = model.to_yaml() 

More details here https://github.com/keras-team/keras/issues/11020#issuecomment-427638145