如何（快速）从特定点的二维图像中提取双线性插值块？

Question

Update: The original question formulation was a bit unclear. I am not just cropping the image but applying bilinear interpolation during the patches extraction process. (See the paper reference below). That's why the algorithm is a bit more involved than just taking slices.

---

I am trying to train a deep learning model to predict face landmarks following this paper . I need to crop parts of the image that contains face into smaller patches around facial landmarks. For example, if we have the image shown below:

The function should generate N=15 "patches", one patch per landmark:

I have the following naïve implementation build on top of torch tensors:

def generate_patch(x, y, w, h, image):
    c = image.size(0)
    patch = torch.zeros((c, h, w), dtype=image.dtype)
    for q in range(h):
        for p in range(w):
            yq = y + q - (h - 1)/2
            xp = x + p - (w - 1)/2
            xd = 1 - (xp - math.floor(xp))
            xu = 1 - (math.ceil(xp) - xp)
            yd = 1 - (yq - math.floor(yq))
            yu = 1 - (math.ceil(yq) - yq)
            for idx in range(c):
                patch[idx, q, p] = (
                    image[idx, math.floor(yq), math.floor(xp)]*yd*xd + 
                    image[idx, math.floor(yq),  math.ceil(xp)]*yd*xu +
                    image[idx,  math.ceil(yq), math.floor(xp)]*yu*xd +
                    image[idx,  math.ceil(yq),  math.ceil(xp)]*yu*xu
                ).item()
    return patch


def generate_patches(image, points, n=None, sz=31):
    if n is None:
        n = len(points)//2
    patches = []
    for i in range(n):
        x_val, y_val = points[i], points[i + n]
        patch = generate_patch(x_val, y_val, sz, sz, image)
        patches.append(patch)
    return patches

The code does its work but too slowly. I guess because of all these for-loops and separate pixels indexing. I would like to vectorize this code, or maybe find some C-based implementation that could do it faster.

I know there is the extract_patches_2d function from sklearn package that helps to pick random patches from the image. However, I would like to pick the patches from specific points instead of doing it randomly. I guess that I can somehow adapt the aforementioned function, or convert the implementation shown above into Cython/C code but probably someone has already done something like this before.

Could you please advise some alternative to the code shown above, or maybe a proposal on how to make it faster? (Except using several parallel workers).

Answer 1

1) use numpy

2) select patches with index extraction. Example:

Patch=img[0:100,0:100]

3) create 3 dimensional body where in 3rd dimension are patches. [15x15xnumber of patches]

4) do your bilinear int. With numpy for all patches in the same time( insted of one pixel calculate with all pixels in 3rd dimension).

That will increase your processing beyond your imagination

If you dont want to get old waiting for your job to be done forget math module. It has no place in datascience.