如何在 Python 中将 3 维数组（在本例中为滤波器组）与 2 维图像（单色）进行卷积？

Question

I have a function definition that takes in an image that is monochromatic and 2 dimensional, and a filter bank that is a 3 dimensional array (48 2D filters). I need to convolve the two to find the feature vector at each pixel location. How do I do that?

I have tried scipy.ndimage.convolve() but get the error "filter weights array has incorrect shape."

Answer 1

To make things simple, simply loop over the temporal dimension of your filter bank, then apply convolution to the image and each filter within the filter bank. After, stack the results into a 3D matrix. This is actually what I would do for readability.

Suppose your image is stored in img and your filters are stored in filters . img is of size M x N and your filters are of size R x C x D with D being the total number of filters you have.

As you've eluded to using scipy.ndimage.convolve , we can just use that. However, it's possible to use cv2.filter2D too. I'll show you how to use both.

Method #1 - Using scipy.ndimage.convolve

import scipy.ndimage
import numpy as np

outputs = []
D = filters.shape[2]
for i in range(D):
    filt = filters[...,i]
    out = scipy.ndimage.convolve(img, filt)
    outputs.append(out)

outputs = np.dstack(outputs)

The above is straight forward. Create an empty list to store our convolution results, then extract the total number of filters we have. After, we loop over each filter, convolve the image with said filter and append it to the list. We then use numpy.dstack to stack all of the 2D responses together to a 3D matrix.

Method #2 - Using cv2.filter2D

import cv2
import numpy as np

outputs = []
D = filters.shape[2]
for i in range(D):
    filt = filters[...,i]
    filt = filt[::-1, ::-1]
    out = cv2.filter2D(img, -1, filt)
    outputs.append(out)

outputs = np.dstack(outputs)

This is exactly the same as Method #1 with the exception of calling cv2.filter2D instead. Also take note that I had to rotate the kernel by 180 degrees as cv2.filter2D performs correlation and not convolution. To perform convolution with cv2.filter2D , you need to rotate the kernel first prior to running the method. Take note that the second parameter to cv2.filter2D is the output data type of the result. We set this to -1 to say that it will be whatever the input data type is.

---

Note on indexing

If you want to avoid indexing into your filter bank all together and let the for loop do that for you, you can shift the channels around so that the number of filters is the first channel. You can then construct the resulting 3D output matrix by list comprehension:

filters = filters.transpose((2, 0, 1))
outputs = np.dstack([scipy.ndimage.convolve(img, filt) for filt in filters])

Answer 2

You can make the monochrome image a 3D array by either padding zeros or replicating the image itself. The number of such paddings would depend on the depth of convolution kernel. For example, let d be the depth of the convolution kernel and I is your image, then

I_pad = np.empty((I.shape[0], I.shape[1], 0))

# Do this for copying the image across channels
I_pad = [np.concatenate((I_pad, I), axis=-1) for _ in range(d)]

# Do this for zero padding
I_pad = [np.concatenate((I_pad, np.zeros(size(I))), axis=-1) for _ in range(d)]

Then carry out the convolution. Hope it helps