在OpenCV中查找图像的原点像素

Question

In python openCV I am trying to create a GUI where the user has to pick pixels at set y coordinates. I can get the openCV pixel location that I want to set the mouse to, but I have no way of tying that to the overall system pixel which is needed for the win32api.SetCursorPos(). I have tried moving the image window with cv2.moveWindow('label', x, y) and then offsetting the cursor by y+offset, but this is a very inexact solution. Is there any way to find the current system pixel where the image origin pixel resides?

Answer 1

I'm not aware of a way to do this directly with OpenCV (after all, it's meant as convenience for prototyping, rather than a full fledged GUI framework), but since we're on Windows, we can hack it using the WinAPI directly.

NB There's a slight complication -- the callback returns image coordinates, so if scaling is enabled, our precision will be limited, and we have to do some extra work to map the coordinates back to client window coordinates.

---

Let's begin by investigating the window hierarchy create by OpenCV for the image display window. We could investigate the source code, but there's a quicker way, using the Spy++ tool from MSVS.

We can write a simple script to show some random data for this:

import cv2
import numpy as np

WINDOW_NAME = u'image'

img = np.zeros((512, 512), np.uint8)
cv2.randu(img, 0, 256)

cv2.namedWindow(WINDOW_NAME, cv2.WINDOW_NORMAL)    
cv2.imshow(WINDOW_NAME, img)
cv2.waitKey()

When we find this window in Spy++, we can see the following info.

There is the top level window, with a caption equal to the window name we specified, of class Main HighGUI class . This window contains a single child window, with no caption, and of class HighGUI class .

---

The following algorithm comes to mind:

1. Use FindWindow to find the top level window by caption, and get it's window handle.

2. Use GetWindow to get the handle of its child window.

3. Use GetClientRect to get the width and height of the client area (which contains the rendered image).

4. Transform the x and y image-relative coordinates back to client area space. (We need to know the dimensions of the current image to do this, so we will pass the current image as the user parameter of the callback.)

5. Transform the coordinates to screen space using ClientToScreen

---

Sample Script:

import win32gui
from win32con import GW_CHILD
import cv2
import numpy as np

# ============================================================================

def on_mouse(event, x, y, flags, img):
    if event != cv2.EVENT_LBUTTONDOWN:
        return

    window_handle = win32gui.FindWindow(None, WINDOW_NAME)
    child_window_handle = win32gui.GetWindow(window_handle, GW_CHILD)

    (_, _, client_w, client_h) =  win32gui.GetClientRect(child_window_handle)

    image_h, image_w = img.shape[:2]

    real_x = int(round((float(x) / image_w) * client_w))
    real_y = int(round((float(y) / image_h) * client_h))

    print win32gui.ClientToScreen(child_window_handle, (real_x, real_y))

# ----------------------------------------------------------------------------    

def show_with_callback(name, img):
    cv2.namedWindow(name, cv2.WINDOW_NORMAL)    
    cv2.setMouseCallback(name, on_mouse, img)
    cv2.imshow(name, img)
    cv2.waitKey()
    cv2.destroyWindow(name)

# ============================================================================

WINDOW_NAME = u'image'

# Make some test image
img = np.zeros((512, 512), np.uint8)
cv2.randu(img, 0, 256)

show_with_callback(WINDOW_NAME, img)