How to recognize an image in another image?

Question

So I'm fairly new to Java, so I'm not sure that this my method of doing this is actually a good idea, but basically I am trying to check for an instance of an image inside another image. So for my testing to see if this would work, I had a 200x148 jpg , got the bytes from that, then got the bytes from a screenshot of the window, and got the bytes from that, and then compared them.

Now, since normally the first image wouldn't be in that screenshot, I opened it in my photos app and put it in while the program was sleeping (before it took the screenshot). And yes, I can confirm that the first image was in the screenshot by looking at the screenshot. However, when I compare the strings (to check if a String with all of the byte data of image one is in a String with all of the byte data of image two), it turns out negative.

Here's the code that I'm trying to use so far:

public static void main(String[] args) throws IOException, HeadlessException, AWTException, InterruptedException  {
     // Get the first image
     ByteArrayOutputStream baos = new ByteArrayOutputStream();
     ImageIO.write(ImageIO.read(new File("profile.jpg")), "jpg", baos);
     byte[] bytes = baos.toByteArray();

     String bytes1S = "";
     for (int i = 0; i < bytes.length; i++) {
         bytes1S += bytes[i];
     }
     // Give yourself enough time to open the other image
     TimeUnit.SECONDS.sleep(6);
     // Take the screenshot
     BufferedImage image = new Robot().createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
     ImageIO.write(image, "jpg", new File("screenshot.jpg"));
     baos = new ByteArrayOutputStream();
     ImageIO.write(ImageIO.read(new File("screenshot.jpg")), "jpg", baos);
     byte[] bytes2 = baos.toByteArray();
     String bytes2S = "";
     for (int i = 0; i < bytes2.length; i++) {
         bytes2S += bytes2[i];
     }
     // Check if the second String of bytes contains the first String of bytes.
     if (bytes2S.contains(bytes1S))
         System.out.println("Yes");
     else
         System.out.println("No");

}

And for reference, here's the first image, and the screenshot that it took:

First image

Screenshot

What's the reason behind it not detecting the first image in the screenshot, and is there a better way to do this (preferably without another library)?

Answer 1

A brute-force approach is to simply load both images as BufferedImage objects, and then walk through the "main" image, pixel by pixel, and see if the "sub image" can be found there.

I have implemented this a while ago, and will post the code below as a MCVE.

But note : When you save images as JPG files, then they are compressed, and this compression is lossy . This means that the pixels will not have the perfectly same colors, even if they have been equal on the screen. In the example below, this is solved pragmatically, with a "threshold" that defines how different the pixels may be. But this is a bit arbitrary and not so reliable. (A more robust solution would require more effort).

I'd strongly recommend to save the images as PNG files. They use a lossless compression. So for PNG files, you can set threshold=0 in the code below.

import java.awt.Color;
import java.awt.Graphics;
import java.awt.Point;
import java.awt.image.BufferedImage;
import java.net.URL;
import java.util.function.IntBinaryOperator;

import javax.imageio.ImageIO;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

public class FindImageInImage
{
    public static void main(String[] args) throws Exception
    {
        BufferedImage mainImage = 
            ImageIO.read(new URL("https://i.stack.imgur.com/rEouF.jpg"));
        BufferedImage subImage = 
            ImageIO.read(new URL("https://i.stack.imgur.com/wISyn.jpg"));

        int threshold = 100;
        Point location = findImageLocation(
            mainImage, subImage, threshold);
        System.out.println("At " + location);

        SwingUtilities.invokeLater(() -> showIt(mainImage, subImage, location));
    }

    private static void showIt(
        BufferedImage mainImage, BufferedImage subImage, Point location)
    {
        JFrame f = new JFrame();
        f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

        f.getContentPane().add(new JPanel()
        {
            @Override
            protected void paintComponent(Graphics g)
            {
                super.paintComponent(g);
                g.drawImage(mainImage, 0, 0, null);
                if (location != null)
                {
                    g.setColor(Color.RED);
                    g.drawRect(location.x, location.y, 
                        subImage.getWidth(), subImage.getHeight());
                }
            }
        });
        f.setSize(1500, 800);
        f.setLocationRelativeTo(null);
        f.setVisible(true);
    }


    static Point findImageLocation(
        BufferedImage mainImage, 
        BufferedImage subImage, 
        int threshold)
    {
        return findImageLocation(mainImage, subImage, (rgb0, rgb1) -> 
        {
            int difference = computeDifference(rgb0, rgb1);
            if (difference > threshold)
            {
                return 1;
            }
            return 0;
        });
    }

    private static int computeDifference(int rgb0, int rgb1)
    {
        int r0 = (rgb0 & 0x00FF0000) >> 16;
        int g0 = (rgb0 & 0x0000FF00) >> 8;
        int b0 = (rgb0 & 0x000000FF);

        int r1 = (rgb1 & 0x00FF0000) >> 16;
        int g1 = (rgb1 & 0x0000FF00) >> 8;
        int b1 = (rgb1 & 0x000000FF);

        int dr = Math.abs(r0 - r1);
        int dg = Math.abs(g0 - g1);
        int db = Math.abs(b0 - b1);

        return dr + dg + db;
    }

    static Point findImageLocation(
        BufferedImage mainImage, 
        BufferedImage subImage, 
        IntBinaryOperator rgbComparator)
    {
        int w = mainImage.getWidth();
        int h = mainImage.getHeight();
        for (int x=0; x < w; x++)
        {
            for (int y = 0; y < h; y++)
            {
                if (isSubImageAt(mainImage, x, y, subImage, rgbComparator))
                {
                    return new Point(x, y);
                }
            }
        }
        return null;
    }

    static boolean isSubImageAt(
        BufferedImage mainImage, int x, int y, 
        BufferedImage subImage, 
        IntBinaryOperator rgbComparator)
    {
        int w = subImage.getWidth(); 
        int h = subImage.getHeight();
        if (x + w > mainImage.getWidth())
        {
            return false;
        }
        if (y + h > mainImage.getHeight())
        {
            return false;
        }
        for (int ix=0; ix < w; ix++)
        {
            for (int iy = 0; iy < h; iy++)
            {
                int mainRgb = mainImage.getRGB(x + ix, y + iy);
                int subRgb = subImage.getRGB(ix, iy);
                if (rgbComparator.applyAsInt(mainRgb, subRgb) != 0)
                {
                    return false;
                }
            }
        }
        return true;
    }

}