OpenCV - Horizontally Flip a Surface Normal Image

Question

I have a collection of Surface Normal images which are like this:

I want to flip them horizontally but since flipping does not account for the change to Surface Normal, I end up with this flipped surface normals which are wrong:

The coloring in the flipped normals is wrong since the surfaces facing left should have a pink coloring and the surfaces faces right should have a blue coloring. ALSO the surfaces facing upwards or downwards should stay the same (green facing upwards and purple facing downwards).

How can I flip the surface normals horizontally correctly? Thanks.

EDIT: it seems I should invert the color but doing so also inverts the upwards and downwards normals making the roofs green and grounds purple (they should stay the same when flipping horizontally). Here's an image of an Inverted Surface Normal (Which seems partly correct and partly incorrect):

Answer 1

Thanks to a Reddit post I was able to solve it. All you have to do is inverting the Red Color Channel after flipping the image.