How to get sub image coordinates?

Question

I have a screenshot of the screen where I need to find the coordinate of the button's center (approximately). Screenshot and sample buttons in *.png format. I'm assuming a method with this signature:

public Coordinate getBtnCoordinate(BufferedImage src, BufferedImage dst) {
    ...
}

@Data
class Coordinate {
    private int x;
    private int y;
}

In the future this will be used in this way:

Coordinate сoordinate = getBtnCoordinates(...);
Robot robot = new Robot();
robot.mouseMove(сoordinate.getX(), сoordinate. getY());
robot.mousePress(InputEvent.BUTTON1_MASK);

But my attempts to implement getBtnCoordinates do not lead to anything for almost a week (((. Help me please implement this method. I will be grateful for any help.

Answer 1

The implementation depends on whether the searched button matches one of the example images exactly or if you need to find "similar" buttons, which is probably a lot harder.

If you search for an exact match, you could scan the screenshot for a pixel or that matches a pixel color at a given position in the button. Once you find a matching pixel, compare the other pixels until there is too much mismatch or the whole button matches.

If the screenshots have a single color background in the interesting area, you can use segmentation, ie find rectangular areas with content first, then compare only these areas to your example buttons.

If the buttons might vary in size, you probably want to segment the buttons themselves into content and top, left, right and bottom border to be able to do partial matches.

If the matches need to be really fuzzy, you may need more advanced techniques, eg use machine learning to train a model of the example buttons.

Without seeing typical example screenshots and buttons for your case, it's hard to provide more concrete advice for this problem.