How to interpret multi-class deep learning classifier by using SHAP?

Question

I trained a multi-class classifier in Keras on IRIS data set. I want to interpret my deep learning model by using SHAP . I use the following lines of code where model is my trained neural network classifier, X1[train] is my training input, and X1[test] is my test input:

import shap
explainer = shap.DeepExplainer(model, X1[train])
shap_values = explainer.shap_values(X1[test])

However, I get this error (I am fine when my softmax has one output in binary classification. The problem appears when softmax has more than 1 output ):

ValueError: Unexpectedly found an instance of type `<class 'numpy.ndarray'>`. Expected a symbolic tensor instance.

During handling of the above exception, another exception occurred:
Layer sequential_96 was called with an input that isn't a symbolic tensor. Received type: <class 'numpy.ndarray'>. Full input:

How can I solve this problem and get SHAP values for a single class in multi class deep learning classifier?

Answer 1

How are you importing keras? When doing from keras import... I had the same issue, but changing to from tensorflow.keras import ... the DeepExplainer worked