Tensorflow model层连接失败，无法使用shap.DeepExplainer

Question

I am trying to apply shap.deepexplainer to explain the model output.

My model class is as follows:

class MyModel(tf.keras.Model):
    def __init__(self,
                 input_dim,
                 emb_dim=128,
                 alpha_hidden_dim_size=128,
                 beta_hidden_dim_size=128,
                 keep_prob_emb=0.25,
                 keep_prob_context=0.25,
                 num_class=1):
        super(MyModel, self).__init__()

        self.embedding = layers.Dense(emb_dim,
                                      use_bias=False,
                                      input_shape=(input_dim, ))

        self.emb_drp = layers.Dropout(keep_prob_emb)

        self.enroll = layers.Dense(emb_dim, activation='tanh')

        
        self.gru_alpha = layers.Bidirectional(
            layers.LSTM(alpha_hidden_dim_size, return_sequences=True))
        self.gru_beta = layers.Bidirectional(
            layers.LSTM(beta_hidden_dim_size, return_sequences=True))

        self.alpha = layers.Dense(1)
        self.beta = layers.Dense(emb_dim, activation='tanh')

        self.context_drp = layers.Dropout(keep_prob_context)

        self.out = layers.Dense(num_class)

    def call(self, visits, enroll, lengths, **kwargs):
        max_len = lengths[tf.argmax(lengths)]
        visits = visits[:, :max_len]
        emb = self.embedding(visits)
        emb = self.emb_drp(emb, training=kwargs.get('training', False))
        enroll = self.enroll(enroll)
        mask = tf.sequence_mask(lengths)
        h_a = self.gru_alpha(emb, mask=mask)
        h_b = self.gru_beta(emb, mask=mask)
        preAlpha = self.alpha(h_a)
        preAlpha = tf.keras.backend.squeeze(preAlpha, axis=2)
        mask_norm = (1 - tf.cast(mask, tf.float32)) * NEG_INF
        alpha = tf.nn.softmax(preAlpha + mask_norm, axis=1)
        beta = self.beta(h_b)
        c_t = tf.math.reduce_sum(alpha[:, :, None] * beta * emb, axis=1)
        c_t = layers.add([c_t, enroll])
        c_t = self.context_drp(c_t, training=kwargs.get('training', False))
        preY = self.out(c_t)

        return preY, alpha, beta

When I applied my model as:

model = MyModel(**flags)

And the model is successfully loaded:

print(model) 
<__main__.MyModel object at 0x7f51db414400>

Then I am trying to use the

background = X.loc[10:20]
e = shap.DeepExplainer((model.layers[0].input, model.layers[-1].output), background)

but then it gives me the error:

AttributeError: Layer dense is not connected, no input to return. Traceback (most recent call last): File "/home/ANANT/codes/test/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py",line 1808, in input' is not connected, no input to return.')

And also the model.layers[-1].output can not give proper output neither:

AttributeError: Layer dense_4 has no inbound nodes. Traceback (most recent call last): File "/home/ANANT/test/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1827, in output raise AttributeError('Layer ' + self.name + ' has no inbound nodes.')

My package versions are: keras==2.3.1, tensorflow==1.15.3, shap==0.35.0

I stuck at this question for a few days, tried shap.KernelExplainer as well, and it gives me a different error:

shap.KernelExplainer(model, df_fis, link="logit")

And the error is as follows:

TypeError: call() missing 2 required positional arguments: 'enroll' and 'lengths' Traceback (most recent call last): File "/home/ANANT/test/env/lib/python3.6/site-packages/shap/explainers/kernel.py", line 97, in __init__model_null = match_model_to_data(self.model, self.data) File "/home/ANANT/test/env/lib/python3.6/site-packages/shap/common.py", line 89, in match_model_to_dataout_val = model.f(data.data) File "/home/ANANT/test/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__outputs = self.call(cast_inputs, *args, **kwargs)

Please help, thanks in advance!

Answer 1

I think you missed the softmax part

Pytorch version

self.softmax = LogSoftmax(dim=1)

Keras version

layers.Dense(num_classes, activation="softmax")

Add the above line at the end of your __init__ method, see if it works