How to input data into external GUI using Java runtime exec

Question

I am trying to write java code to access and use the DullRazor software.

Please refer to this image of the DullRazor application:

I had an idea of creating a Java runtime program that could loop through all images I need to process(the software only allows 1 image at a time) and complete the necessary steps required for the DullRazor software to successfully alter an image for every image I have.

The DullRazor software works as follows:

-Source File: Requires the path to an image(jpg in my case) to be altered ie c://Isic-Images//image0000.jpg.

-Target File: Requires the location for the new image with a new image name ie c://finalLocation//newImage.jpg

-Start: Run the program after giving the inputs in the correct format as described above.

My thinking is iterating through all my images, creating new ones and incrementing the name(img00, img001 etc..).

I have never attempted anything like this in Java and I have had some trouble accessing the Input fields of the software as well as the application's start button.

The code below is just very basic for opening the application, but I am unsure how to access the various items in the DullRazor application and being able to input Strings in those aforementioned fields(again, refer to the DullRazor picture).

private String trainingPath = "C:\\Users\\user\\AppData\\Local\\Temp\\ISIC-Images\\Training\\0";
private String finalPath = "C:\\Users\\user\\finalLocation\\";

public static void main(String[] args) {


try {
    Runtime runTime = Runtime.getRuntime();
    Process process = runTime.exec("C:\\Users\\user\\Desktop\\dullrazor.exe");
    System.out.println("Opening DullRazor");
    OutputStream output = process.getOutputStream();
    InputStream input = process.getInputStream();
    Thread.sleep(2000);
    process.destroy();
} catch (InterruptedException e) {
    e.printStackTrace();
} catch (IOException s) {
    s.printStackTrace();
} finally {
    System.out.println("Closing Dullrazor");
}

}

I have just been testing a bit with the code above, but I am unsure on how to proceed.

Tell me if there is anything that needs clarifying.

Any help is greatly appreciated, thanks.

Answer 1

You can use Java's java.awt.Robot class to control mouse and keyboard on the screen.

This is a simple example entering "test1" and "test2" into two input fields:

Robot r = new Robot();

r.mouseMove(22, 125);
r.mousePress(InputEvent.BUTTON1_DOWN_MASK);
r.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);

r.keyPress('T');
r.keyRelease('T');
r.keyPress('E');
r.keyRelease('E');
r.keyPress('S');
r.keyRelease('S');
r.keyPress('T');
r.keyRelease('T');
r.keyPress('1');
r.keyRelease('1');

r.mouseMove(200, 125);
r.mousePress(InputEvent.BUTTON1_DOWN_MASK);
r.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);

r.keyPress('T');
r.keyRelease('T');
r.keyPress('E');
r.keyRelease('E');
r.keyPress('S');
r.keyRelease('S');
r.keyPress('T');
r.keyRelease('T');
r.keyPress('2');
r.keyRelease('2');

The above code in action:

If the position of the new application window does not change with each start, and the tool is not about to be deployed to users, this might already suffice. However, if it changes the position with each start, the challenge is to find the window position to add the relative input element positions from there. There are Windows (platform) specific approaches facilitating the Win32 API through JNA, though I'm not familiar with it and whether it is still available in current Microsoft Windows versions.

See these related questions on determining other windows positions:

• Windows: how to get a list of all visible windows?
• How to get the x and y of a program window in Java?

Answer 2

Using robot works perfectly in order to input into the targeted fields and clicking start/clear button on the application.

In order to find the x & y positions of the application I used runtime exec to open dullrazor and then take a screenshot of the screen with the application up where mouse clicks reveals the x and y position of the current click. Below is the code for finding x & y which I found at this Stackoverflow thread :

Robot robot = new Robot();
    final Dimension screenSize = Toolkit.getDefaultToolkit().
        getScreenSize();
    final BufferedImage screen = robot.createScreenCapture(
        new Rectangle(screenSize));

    SwingUtilities.invokeLater(new Runnable() {
        public void run() {
            JLabel screenLabel = new JLabel(new ImageIcon(screen));
            JScrollPane screenScroll = new JScrollPane(screenLabel);
            screenScroll.setPreferredSize(new Dimension(
                (int)(screenSize.getWidth()/2),
                (int)(screenSize.getHeight()/2)));

            final Point pointOfInterest = new Point();

            JPanel panel = new JPanel(new BorderLayout());
            panel.add(screenScroll, BorderLayout.CENTER);

            final JLabel pointLabel = new JLabel(
                "Click on any point in the screen shot!");
            panel.add(pointLabel, BorderLayout.SOUTH);

            screenLabel.addMouseListener(new MouseAdapter() {
                public void mouseClicked(MouseEvent me) {
                    pointOfInterest.setLocation(me.getPoint());
                    pointLabel.setText(
                        "Point: " +
                        pointOfInterest.getX() +
                        "x" +
                        pointOfInterest.getY());
                }
            });

            JOptionPane.showMessageDialog(null, panel);

            System.out.println("Point of interest: " + pointOfInterest);
        }
    });

Thank you try-catch-finally for a great answer.