没有 CNN 的重复模式的图像语义分割

Question

Suppose I have one or multiple tiles consisting of a single pattern (eg materials like: wood, concrete, gravel...) that I would like to train my classifier on, and then I'll use the trained classifier to determine to which class each pixel in another image belong.

Below are example of two tiles I would like to train the classifier on:

And let's say I want to segment the image below to identify the pixels belonging to the door and those belonging to the wall. It's just an example, I know this image isn't made of exactly the same patterns as the tiles above:

For this specific problem, is it necessary to use convolutional neural networks? Or is there a way to achieve my goal with a shallow neural network or any other classifier, combined with texture features for example?

I've already implemented a classifier with Scikit-learn which works on tile pixels individually (see code below where training_data is a vector of singletons), but I want instead to train the classifier on texture patterns.

# train classifier
classifier = SGDClassifier()
classifier.fit(training_data, training_target)

# classify given image
test_data = image_gray.flatten().reshape((-1, 1))
predictions = classifier.predict(test_data)
image_classified = predictions.reshape(image_gray.shape)

I was reading this review of recent deep learning methods used for image segmentation and the results seem accurate, but since I've never used any CNN before I feel intimidated by it.

Answer 1

Convolutional Neural Networks (CNNs) are high performance tools for image recognition (including semantic segmentation) and have been shown to be very sensitive to texture . The field of computer vision has been around way before the current wave of interest in deep learning, however, and there are various other tools that are still relevant - often with smaller requirements for computational resources and/or training data.

For this specific problem, is it necessary to use convolutional neural networks?

It very much depends on what your metrics for success are. There are other tools that do not involve the use of CNNs - whether they will give you a satisfactory level of detection accuracy can only be determined by practical testing.

Or is there a way to achieve my goal with a shallow neural network or any other classifier, combined with texture features for example?

A shallow neural network will have some detection capability, although (unlike CNNs), they do not exhibit translational invariance and so are sensitive to small displacements of the target. Such a network is likely to have more success if used to classify small patches of the image; classifying an image patch within a sliding window is not that unlike how a CNN works, of course. It is also possible to approximate a CNN using an equivalent multi-layer perceptron (MLP) - that would be another approach, if your definition of 'shallow' permits.

Two approaches that do not require neural networks:

Histogram of Oriented Gradients The HOG descriptor extracts image features using a histogram of gradients in the horizontal and vertical axis. This produces a feature vector, that can be classified - such as using a Support Vector Machine (SVM) or shallow neural network (MLP), for example. This would be a viable approach to classifying image patches without using CNNs. The scikit-image package has a HOG function , and there is a full worked example of classification of HOG features here . From the documentation:

from skimage.feature import hog
from skimage import data, exposure

image = data.astronaut()

fd, hog_image = hog(image, orientations=8, pixels_per_cell=(16, 16),
                    cells_per_block=(1, 1), visualize=True, multichannel=True)

Felsenszwalb's efficient graph based image segmentation There are a bunch of segmentation algorithms in the scikit-image.segmentation toolbox. Felsenszwalb's is one of them, which (broadly speaking) performs a clustering of image regions based on edges. More info here . From the module documentation:

from skimage.segmentation import felzenszwalb
from skimage.data import coffee
img = coffee()
segments = felzenszwalb(img, scale=3.0, sigma=0.95, min_size=5)

Hope that helps.

Answer 2

You can use U-Net or SegNet for image segmentation. In fact you add residual layers to your CNN to get this result:

About U-Net :

Arxiv: U-Net: Convolutional Networks for Biomedical Image Segmentation

Seg-Net :

Arxiv: SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

Here are Simple Examples of Codes: keras==1.1.0

U-Net:

shape=60
batch_size = 30
nb_classes = 10
img_rows, img_cols = shape, shape
nb_filters = 32
pool_size = (2, 2)
kernel_size = (3, 3)
input_shape=(shape,shape,1)

reg=0.001
learning_rate = 0.013
decay_rate = 5e-5
momentum = 0.9

sgd = SGD(lr=learning_rate,momentum=momentum, decay=decay_rate, nesterov=True)
shape2

recog0 = Sequential()
recog0.add(Convolution2D(20, 3,3,
                        border_mode='valid',
                        input_shape=input_shape))
recog0.add(BatchNormalization(mode=2))

recog=recog0
recog.add(Activation('relu'))
recog.add(MaxPooling2D(pool_size=(2,2)))
recog.add(UpSampling2D(size=(2, 2)))
recog.add(Convolution2D(20, 3, 3,init='glorot_uniform'))
recog.add(BatchNormalization(mode=2))
recog.add(Activation('relu'))

for i in range(0,2):
    print(i,recog0.layers[i].name)

recog_res=recog0
part=1
recog0.layers[part].name
get_0_layer_output = K.function([recog0.layers[0].input, K.learning_phase()],[recog0.layers[part].output])

get_0_layer_output([x_train, 0])[0][0]

pred=[np.argmax(get_0_layer_output([x_train, 0])[0][i]) for i in range(0,len(x_train))]

loss=x_train-pred
loss=loss.astype('float32')

recog_res.add(Lambda(lambda x: x,input_shape=(56,56,20),output_shape=(56,56,20)))
recog2=Sequential()
recog2.add(Merge([recog,recog_res],mode='ave'))
recog2.add(Activation('relu'))
recog2.add(Convolution2D(20, 3, 3,init='glorot_uniform'))
recog2.add(BatchNormalization(mode=2))
recog2.add(Activation('relu'))
recog2.add(Convolution2D(1, 1, 1,init='glorot_uniform'))
recog2.add(Reshape((shape2,shape2,1)))
recog2.add(Activation('relu'))

recog2.compile(loss='mean_squared_error', optimizer=sgd,metrics = ['mae'])
recog2.summary()

x_train3=x_train2.reshape((1,shape2,shape2,1))

recog2.fit(x_train,x_train3,
                nb_epoch=25,
                batch_size=30,verbose=1)

SegNet:

shape=60
batch_size = 30
nb_classes = 10
img_rows, img_cols = shape, shape
nb_filters = 32
pool_size = (2, 2)
kernel_size = (3, 3)
input_shape=(shape,shape,1)

reg=0.001
learning_rate = 0.012
decay_rate = 5e-5
momentum = 0.9

sgd = SGD(lr=learning_rate,momentum=momentum, decay=decay_rate, nesterov=True)

recog0 = Sequential()
recog0.add(Convolution2D(20, 4,4,
                        border_mode='valid',
                        input_shape=input_shape))
recog0.add(BatchNormalization(mode=2))
recog0.add(MaxPooling2D(pool_size=(2,2)))

recog=recog0
recog.add(Activation('relu'))
recog.add(MaxPooling2D(pool_size=(2,2)))
recog.add(UpSampling2D(size=(2, 2)))
recog.add(Convolution2D(20, 1, 1,init='glorot_uniform'))
recog.add(BatchNormalization(mode=2))
recog.add(Activation('relu'))

for i in range(0,8):
    print(i,recog0.layers[i].name)

recog_res=recog0
part=8
recog0.layers[part].name
get_0_layer_output = K.function([recog0.layers[0].input, K.learning_phase()],[recog0.layers[part].output])
get_0_layer_output([x_train, 0])[0][0]
pred=[np.argmax(get_0_layer_output([x_train, 0])[0][i]) for i in range(0,len(x_train))]

loss=x_train-pred
loss=loss.astype('float32')

recog_res.add(Lambda(lambda x: x-np.mean(loss),input_shape=(28,28,20),output_shape=(28,28,20)))

recog2=Sequential()
recog2.add(Merge([recog,recog_res],mode='sum'))
recog2.add(UpSampling2D(size=(2, 2)))
recog2.add(Convolution2D(1, 3, 3,init='glorot_uniform'))
recog2.add(BatchNormalization(mode=2))
recog2.add(Reshape((shape2*shape2,)))
recog2.add(Reshape((shape2,shape2,1)))
recog2.add(Activation('relu'))
recog2.compile(loss='mean_squared_error', optimizer=sgd,metrics = ['mae'])
recog2.summary()

x_train3=x_train2.reshape((1,shape2,shape2,1))

recog2.fit(x_train,x_train3,
                nb_epoch=400,
                batch_size=30,verbose=1)

Then add a threshold for the colors of segmentation.