如何使用Appium在移动应用程序中获取烤面包机消息的文本

Question

I am trying to verify a toaster message in Android Mobile app but not able to get text of toaster message as it doesn't show in uiautomatorviewer. Got some information that by the help of OCR it can be done taking screenshots and fetching the text from that screenshots Can anyone help me out how to do this step by step using java in Appium project?

Answer 1

You can follow the information on the below links to install the Tesseract on your machine:

For Mac: http://emop.tamu.edu/Installing-Tesseract-Mac

For Windows: http://emop.tamu.edu/Installing-Tesseract-Windows8

After installing the TessEract on your machine you need to add the dependency of TessEract Java library in your project. If you are using Maven for it, adding below dependency will work:

    <dependency>
        <groupId>org.bytedeco.javacpp-presets</groupId>
        <artifactId>tesseract</artifactId>
        <version>3.04-1.1</version>
    </dependency>

Also the 'Step 3' which is mentioned by Ivan need not to be followed.

If you are using 'TestNG' the TessEract API needs to be initialised only once so instead of initialising it every time, as per your framework you can initialise it either in the 'BeforeTest' or 'BeforeSuite' or 'BeforeClass' method and accordingly close the API either in 'AfterTest' or 'AfterSuite' or 'AfterClass' method.

Below is the code that I have written to achieve it.

import static org.bytedeco.javacpp.lept.pixDestroy;
import static org.bytedeco.javacpp.lept.pixRead;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
import org.bytedeco.javacpp.lept.PIX;
import org.bytedeco.javacpp.tesseract.TessBaseAPI;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;

    public class BaseTest {

    static TessBaseAPI api = new TessBaseAPI();

    @BeforeSuite
        public void beforeSuit() throws IOException {

            File screenshotsDirec = new File("target/screenshots");

            if (screenshotsDirec.exists())
                FileUtils.forceDelete(screenshotsDirec);

            FileUtils.forceMkdir(screenshotsDirec);

            System.out.println("Initializing TessEract library");

            if (api.Init("/opt/local/share", "eng") != 0) {
                System.err.println("Could not initialize tesseract.");
            }

        }

    public synchronized boolean verifyToastMessage(String msg)
                throws IOException {
            TakesScreenshot takeScreenshot = ((TakesScreenshot) driver);

            File[] screenshots = new File[5];

            for (int i = 0; i < screenshots.length; i++) {
                screenshots[i] = takeScreenshot.getScreenshotAs(OutputType.FILE);
            }

            String outText;

            Boolean isMsgContains = false;

            for (int i = 0; i < screenshots.length; i++) {
                PIX image = pixRead(screenshots[i].getAbsolutePath());
                api.SetImage(image);
                outText = api.GetUTF8Text().getString().replaceAll("\\s", "");
                System.out.println(outText);
                isMsgContains = outText.contains(msg);
                pixDestroy(image);
                if (isMsgContains) {
                    break;
                }
            }

            return isMsgContains;

        }

    @AfterSuite()
        public void afterTest() {

            try {
                api.close();
            } catch (Exception e) {
                api.End();
                e.printStackTrace();
            }

        }
    }

I would also like to add that writing tests to read and verify the Toast messages is this way is not very much reliable as in one of my tests this code successfully captures the Toast message while in another test it fails to capture the toast message because the capturing of the screenshots starts when the toast message disappears. That was the reason I tried to write this code very much efficiently. However that also does not serve the purpose.

Answer 2

Follow this discussion on Appium forum: https://discuss.appium.io/t/verifying-toast/3676 .

Basic steps to verify a Toaster are:

1. Perform action to trigger the toast message to appear on screen
2. Take x number of screenshots
3. Increase resolutions of all screenshots
4. Use tessearct OCR to detect the toast message.

Refer this repo to use Java OCR library (see at the bottom):

import org.bytedeco.javacpp.*;
import static org.bytedeco.javacpp.lept.*;
import static org.bytedeco.javacpp.tesseract.*;

public class BasicExample {
    public static void main(String[] args) {
        BytePointer outText;

        TessBaseAPI api = new TessBaseAPI();
        // Initialize tesseract-ocr with English, without specifying tessdata path
        if (api.Init(null, "eng") != 0) {
            System.err.println("Could not initialize tesseract.");
            System.exit(1);
        }

        // Open input image with leptonica library
        PIX image = pixRead(args.length > 0 ? args[0] : "/usr/src/tesseract/testing/phototest.tif");
        api.SetImage(image);
        // Get OCR result
        outText = api.GetUTF8Text();
        System.out.println("OCR output:\n" + outText.getString());

        // Destroy used object and release memory
        api.End();
        outText.deallocate();
        pixDestroy(image);
    }
}