尝试使用 grad-CAM 热图可视化 ML，但 K.gradients 返回零值

Question

for visualizing the results of the ML process i decided to use a CAM heatmap. I used the code provided here on my model yet the return of pooled_grads_value, conv_layer_output_value = iterate([x]) is all zeros for pooled_grads_value but the conv_layer_output_value has values in it.

this is how i used it:

import matplotlib.image as mpimg
from keras import backend as K
import matplotlib.pyplot as plt
import pandas as pd
import keras
import numpy as np
#%%
model_path = "pathto/model.h5"
img_path = "pathto/pic.tif"
output_path = "pathto/output.jpeg"
size = (1000,200)
#%%

model = keras.models.load_model(model_path)

img=mpimg.imread(img_path)
plt.imshow(img)
#%%

from keras.preprocessing import image
img = image.load_img(img_path, target_size=size)
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)

#%%
preds = model.predict(x)

argmax = np.argmax(preds[0])

output = model.output[:, argmax]
#%%
last_conv_layer = model.get_layer( "conv_6_l")
grads = K.gradients(output, last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
#%%
pooled_grads_value,  conv_layer_output_value = iterate([x])
#%%
for i in range(64):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]
#%%
heatmap = np.mean(conv_layer_output_value, axis=-1)
heatmap = np.maximum(heatmap, 0)
#heatmap /= np.max(heatmap)
#%%
import cv2
img = cv2.imread(img_path)
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
hif = .8
superimposed_img = heatmap * hif + img
cv2.imwrite("pathto/justheatmap.jpeg", heatmap * hif)
#%%
cv2.imwrite(output_path, superimposed_img)

my model looks like this:

model = Sequential()
model.add(Conv2D(8, (3, 3), input_shape=(1000, 200, 3), name = "conv_1"))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(16, (3, 3), name="conv_2"))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3), name="conv_3"))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3),name="conv_4"))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(128, (3, 3),name="conv_5"))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3),name="conv_6_l"))
model.add(Activation('relu'))

model.add(GlobalAveragePooling2D())

model.add(Dropout(0.33))
model.add(Dense(5))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
          optimizer='Adam',
          metrics=['accuracy'])

Does anyone know what the source of my problem is?

I also tried a different aproach shown here using tensorflow.keras.GradientTape() function , yet this gives me an error module 'tensorflow' has no attribute 'GradientTape so I sticked to the first aproach.

I'm using keras V 2.3.1 and tensorflow V 2.1.0 in python.

I also uploaded a zip-file containing the .h5 file of my model and one picture for testing

Thanks in advance for everyone trying to help!

Hans

Answer 1

We can capture Gradients using Keras Backend OR Tensorflow tf.GradientTape() function. Later we can use these Gradients to visualize as Heatmap. Would like to route you to two SO Answers that have explained on how to capture the gradients and visualize them as Heatmap. These 2 SO Answers would suffice your requirements.

Please refer to this SO Answer for capturing the gradients using Keras Backend and also for visualizing the gradients as heat map.

If you are keen on capturing the gradients using the Tensorflow tf.GradientTape() function then please refer to this SO Answer and for visualizing the Gradients as heatmap you can use the same code mentioned in the earlier answer.