How can I compute the location of a specific point after a mesh transformation in Python PIL (Pillow)?

Question

If I transform image using:

This code was taken from here: https://stackoverflow.com/a/37350619/259757

im_transformed = im.transform(im.size, Image.MESH, mesh)

Given a specific coordinate (x,y) from my original image how can I compute what the new coordinates would be in im_transformed? Basically could I apply the same transformation to a specific single point like is possible with affine transformations?

My reason for wanting to do this is that lets say I have the location known of a smily face in the image. In the second image that has been transformed/warped I want to make a prediction where where that smily face is and compare the prediction to where it should be. I know its not exact but the idea is that I can generate clean image with known locations of smily face, warp them and have a new location in x,y where the smily face should be in the warped image. This will allow me to generate training data for a machine learning model.

With an affine transformation this is pretty straightforward but with a mesh transformation I do not know how to do this.

Answer 1

So after a little digging in PIL code this is what i could find:

A MESH transform is just a multiple application of QUAD transforms.

Each QUAD transform maps a quadrilateral in the source image to a rectangle in the destination.

So lets see what a QUAD transform is: PIL quad transform

   quad_transform(double* xin, double* yin, int x, int y, void* data) {

    double* a = (double*) data;
    double a0 = a[0]; double a1 = a[1]; double a2 = a[2]; double a3 = a[3];
    double a4 = a[4]; double a5 = a[5]; double a6 = a[6]; double a7 = a[7];

    xin[0] = a0 + a1*x + a2*y + a3*x*y;
    yin[0] = a4 + a5*x + a6*y + a7*x*y;

    return 1;
}

This is an inverse mapping that tells how to sample a point in the source image to get the pixel in the location xin,yin in the target image.

The parameters are calculated in the following way and i rewrote it so it looks nicer:

def get_parameters(rectangle, quadrilateral):
 w = rectangle[2] - rectangle[0]
 h = rectangle[3] - rectangle[1]

 nw = quadrilateral[0:2]  # The quadrilateral coordinates
 sw = quadrilateral[2:4]  # The quadrilateral coordinates
 se = quadrilateral[4:6]  # The quadrilateral coordinates
 ne = quadrilateral[6:8]  # The quadrilateral coordinates
 x0, y0 = nw
 As = 1.0 / w
 At = 1.0 / h
 parameters = (
    x0,  # a0
    (ne[0] - x0) * As,  # a1,
    (sw[0] - x0) * At,  # a2
    (se[0] - sw[0] - ne[0] + x0) * As * At,  # a3
    y0,  # a4
    (ne[1] - y0) * As,  # a5
    (sw[1] - y0) * At,  # a6
    (se[1] - sw[1] - ne[1] + y0) * As * At,  # a7
 )
 return parameters

The problem is that you want the transformation from the point in the QUAD to the resulted point in the rectangle.

So you need to solve the following equations for x and y (not for x_rect and y_rect:!!!):

x_rect = a0 + a1*x + a2*y + a3*x*y;
y_rect = a4 + a5*x + a6*y + a7*x*y;

This inversion is not the nicest and gives you a quadratic equation in x and y so you will need to decide which solution to take according to the resulting coordinates.

For a full explanation please look at Answer