预测和类激活图仅在某些图片上起作用

Question

I have retrained a VGG16 classifier and want to show the class activation map. Unfortunately this only works with some pictures, despite the images are preprocessed. It is only a binary classifier.

I have seen that some pictures are not in the desired width and height, despite setting the target_size while loading the image. Manual resizing did not help as well. z has the desired shape.

from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input, decode_predictions
from keras.engine.input_layer import Input
from keras.layers import Dropout, Flatten, Dense
from keras.preprocessing import image
from keras.models import load_model, Model
from keras import backend as K
import numpy as np
import matplotlib.pyplot as plt
import cv2

# Load weights from retrained classifier
top_model_weights_path = 'retrained_weights.h5'

# Create model with VGG16 base
input_tensor = Input(shape=(224, 224, 3))
base_model = VGG16(weights='imagenet', include_top=False, input_tensor=input_tensor)
x = Flatten()(base_model.output)
x = Dense(4096, activation='relu')(x)
x = Dense(4096, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)
model = Model(input=base_model.input, output=predictions)
model.load_weights(top_model_weights_path, by_name=True)


# load and preprocess image
img_path = './picture.jpg'

img = image.load_img(img_path, target_size=(224, 224))

z = image.img_to_array(img)

z = np.expand_dims(z, axis=0)

z = preprocess_input(z)

# make Prediction
preds = model.predict(z)
print(preds)

maximum_model_output = model.output[:, 0]

last_conv_layer = model.layers[17]

# pooled grads of last convolutional layer and iterate over image
grads = K.gradients(model.output[:, 0], last_conv_layer.output)[0]
pooled_grads = K.mean(grads, axis=(0, 1, 2))
iterate = K.function([model.input],
                     [pooled_grads, last_conv_layer.output[0]])
pooled_grads_value, conv_layer_output_value = iterate([z])

for i in range(512):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

# create heatmap
heatmap = np.mean(conv_layer_output_value, axis=-1)

heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)
plt.matshow(heatmap)

img = cv2.imread(img_path)

heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)

superimposed_img = heatmap * 0.4 + img

cv2.imwrite('./Images/picture_cam.jpg', superimposed_img)

As stated, with some pictures I get a valid prediction and a class activation map, most pictures do not work and I got this error (last_conv_layer.output is all 0, pooled_grads are all 0 and value of model.predict is 1.0):

Using TensorFlow backend.
Backend TkAgg is interactive backend. Turning interactive mode on.
2019-08-11 21:13:16.868637: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
<input>:20: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("in..., outputs=Tensor("de...)`
[[1.]]
0
<input>:63: RuntimeWarning: invalid value encountered in true_divide
/home/zuse/Projekte/deep-learning-convolutional-network/venv/lib/python3.6/site-packages/matplotlib/image.py:395: UserWarning: Warning: converting a masked element to nan.
  dv = (np.float64(self.norm.vmax) -
/home/zuse/Projekte/deep-learning-convolutional-network/venv/lib/python3.6/site-packages/matplotlib/image.py:396: UserWarning: Warning: converting a masked element to nan.
  np.float64(self.norm.vmin))
/home/zuse/Projekte/deep-learning-convolutional-network/venv/lib/python3.6/site-packages/matplotlib/image.py:403: UserWarning: Warning: converting a masked element to nan.
  a_min = np.float64(newmin)
/home/zuse/Projekte/deep-learning-convolutional-network/venv/lib/python3.6/site-packages/matplotlib/image.py:408: UserWarning: Warning: converting a masked element to nan.
  a_max = np.float64(newmax)
/home/zuse/Projekte/deep-learning-convolutional-network/venv/lib/python3.6/site-packages/matplotlib/colors.py:918: UserWarning: Warning: converting a masked element to nan.
  dtype = np.min_scalar_type(value)

I am running out of ideas what could be the problem.

Answer 1

Issue was connected to preprocess_input from applications.vgg16. Setting

 z = preprocess_input(z, mode='tf') 

solved the issue. Perhaps it helps someone.