JNA / WinAPI。 模拟鼠标点击移动鼠标 cursor 并且不会将其返回到开始 position

Question

In my Java code, I would like to simulate invisible mouse click on certain screen coordinates (x, y) when key Q is pressed. It should be invisible for human eye: mouse move -> mouse click -> mouse move back.

I am using JNA 5.6.0 and jna-platform 5.6.0 for this purpose which use native WinAPI functions. I implemented Low level keyboard hook inside LowLevelKeyboardProc() {callback()} that intercepts keystrokes. For mouse click simulation I am using SendInput() .

I was expected that mouse click should be done on certain global screen coordinates (x, y). In my code example, coordinates are 40, 40.

But instead of expected result I got:

• the mouse movement and mouse click but without returning it back to the start position where cursor was located before click.
• mouse moves and makes a click by coordinates 40, 40 from current mouse position and not by the coordinates of the global screen.

The best way to demonstate this behavior, run the code provided below, open Paint, select the brush tool and press 'Q' button.

This is how it looks like:

As you may see cursor doesn't move back to the start position after first time pressing Q button. The next presses of the Q button have more distance between clicked point and current mouse position and this distance differs after first pressing Q button.

import com.sun.jna.Pointer;
import com.sun.jna.platform.win32.*;

import static com.sun.jna.platform.win32.WinUser.*;

public class TestExample {

    public static final int MOUSEEVENTF_MOVE = 1;
    public static final int MOUSEEVENTF_LEFTDOWN = 2;
    public static final int MOUSEEVENTF_LEFTUP = 4;

    private static WinUser.HHOOK hHook;
    static final User32 user32Library = User32.INSTANCE;
    static WinDef.HMODULE hMod = Kernel32.INSTANCE.GetModuleHandle(null);

    public static void main(String[] args) {
        keyReMapOn();
    }

    public static void keyReMapOn() {
        WinUser.LowLevelKeyboardProc keyboardHook = new WinUser.LowLevelKeyboardProc() {
            @Override
            public WinDef.LRESULT callback(int nCode, WinDef.WPARAM wParam, WinUser.KBDLLHOOKSTRUCT kbDllHookStruct) {
                if (nCode >= 0) {
                    if (wParam.intValue() == WM_KEYDOWN) {
                        if (kbDllHookStruct.vkCode == 81) {  // Q button
                            clickByCord(40, 40);            
                            return new WinDef.LRESULT(1);
                        }
                    }
                }
                Pointer ptr = kbDllHookStruct.getPointer();
                long peer = Pointer.nativeValue(ptr);
                return user32Library.CallNextHookEx(hHook, nCode, wParam, new WinDef.LPARAM(peer));
            }
        };

        hHook = user32Library.SetWindowsHookEx(WH_KEYBOARD_LL, keyboardHook, hMod, 0);

        int result;
        WinUser.MSG msg = new WinUser.MSG();
        while ((result = user32Library.GetMessage(msg, null, 0, 0)) != 0) {
            if (result == -1) {
                break;
            } else {
                user32Library.TranslateMessage(msg);
                user32Library.DispatchMessage(msg);
            }
        }
    }

    public static void clickByCord(int x, int y) {
        mouseMove(x, y);
        mouseLeftClick(x, y);
    }

    static void mouseMove(int x, int y) {
        mouseAction(x, y, MOUSEEVENTF_MOVE);
    }

    public static void mouseLeftClick(int x, int y) {
        mouseAction(x, y, MOUSEEVENTF_LEFTDOWN);
        mouseAction(x, y, MOUSEEVENTF_LEFTUP);
    }

    public static void mouseAction(int x, int y, int flags) {
        INPUT input = new INPUT();

        input.type = new DWORD(INPUT.INPUT_MOUSE);
        input.input.setType("mi");
        if (x != -1) {
            input.input.mi.dx = new LONG(x);
        }
        if (y != -1) {
            input.input.mi.dy = new LONG(y);
        }
        input.input.mi.time = new DWORD(0);
        input.input.mi.dwExtraInfo = new BaseTSD.ULONG_PTR(0);
        input.input.mi.dwFlags = new DWORD(flags);
        User32.INSTANCE.SendInput(new DWORD(1), new INPUT[]{input}, input.size());
    }
}

I made mistakes in the code somewhere.

Can anyone tell me how I can achieve invisible left mouse click on the coordinates of the global screen?

Answer 1

The next presses of the Q button have more distance between clicked point and current mouse position and this distance differs after first pressing Q button.

The documentation already explains this:

Relative mouse motion is subject to the effects of the mouse speed and the two-mouse threshold values. A user sets these three values with the Pointer Speed slider of the Control Panel's Mouse Properties sheet. You can obtain and set these values using the SystemParametersInfo function.

The system applies two tests to the specified relative mouse movement . If the specified distance along either the x or y axis is greater than the first mouse threshold value, and the mouse speed is not zero, the system doubles the distance. If the specified distance along either the x or y axis is greater than the second mouse threshold value, and the mouse speed is equal to two, the system doubles the distance that resulted from applying the first threshold test. It is thus possible for the system to multiply specified relative mouse movement along the x or y axis by up to four times.

You can use absolute coordinates to complete the project.

If MOUSEEVENTF_ABSOLUTE value is specified, dx and dy contain normalized absolute coordinates between 0 and 65,535. The event procedure maps these coordinates onto the display surface. Coordinate (0,0) maps onto the upper-left corner of the display surface; coordinate (65535,65535) maps onto the lower-right corner. In a multimonitor system, the coordinates map to the primary monitor.

Part of the code:

UINT mouseAction(int x, int y, int flags)
{
    INPUT input;
    POINT pos;
    GetCursorPos(&pos);

    input.type = INPUT_MOUSE;
    input.mi.dwFlags = flags;
    input.mi.time = NULL; 
    input.mi.mouseData = NULL;
    input.mi.dx = (pos.x + x)*(65536.0f / GetSystemMetrics(SM_CXSCREEN));
    input.mi.dy = (pos.y + y)*(65536.0f / GetSystemMetrics(SM_CYSCREEN));
    input.mi.dwExtraInfo = GetMessageExtraInfo();

    return SendInput(1, &input, sizeof(INPUT));
}

void mouseMove(int x, int y)
{
    mouseAction(x, y, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE);
}

The code is C++, I am not familiar with JAVA code, but I think it is easy enough for you to change to Java.

Supplement: You can use GetAsyncKeyState to detect whether the Q key is pressed. Too many dependent hooks will affect operation performance.

As you may see cursor doesn't move back to the start position after first time pressing Q button.

You can save the starting coordinates. Then return to the starting position when needed.

Gif demo:

Updated:

To further help you solve the problem, I will demonstrate some operations for your reference.

We use GetCursorPos to get the coordinates of ABC.

A(267,337) B(508,334) C(714,329)

Then pass their coordinates to the input structure, refer to the following code:

#include <Windows.h>
#include <iostream>

UINT mouseAction(int x, int y, int flags)
{
    INPUT input;
    POINT pos;
    int x1 = GetSystemMetrics(SM_CXSCREEN);
    int y1 = GetSystemMetrics(SM_CYSCREEN);
    input.type = INPUT_MOUSE;
    input.mi.dwFlags = flags;
    input.mi.time = NULL;
    input.mi.mouseData = NULL;
    input.mi.dx = x* (65536.0f / GetSystemMetrics(SM_CXSCREEN));
    input.mi.dy = y* (65536.0f / GetSystemMetrics(SM_CYSCREEN));
    input.mi.dwExtraInfo = GetMessageExtraInfo();

    return SendInput(1, &input, sizeof(INPUT));
}


int main()
{           
    while(1)
    {
        if (GetAsyncKeyState(0x51) & 0x0001)
        {
            mouseAction(267, 337, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE);
        }
        if (GetAsyncKeyState(0x57) & 0x0001)
        {
            mouseAction(508, 334, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE);
        }
        if (GetAsyncKeyState(0x45) & 0x0001)
        {
            mouseAction(714, 329, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE);
    //        mouseAction(267-87, 337-65, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE);
        }
    }

    return 0;
}

Although it is C++ code, the basic logic is the same as that of winapi. Press the Q key, the mouse will move to the A coordinate, press the W key to move to the B coordinate, and press the E key to move to the C coordinate.

Then we can calculate the coordinate of D based on the distance between A and D, which can help you solve the problem of moving two adjacent coordinates.

In addition, we can fine-tune the distance according to the actual situation.

Add the following code,

mouseAction(267-87, 337-65, MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE); // D(267-87, 337-65)