Appium Image Recognition Using Java – Devstringx

Back to Blog
banner for appium java blog

Appium Image Recognition Using Java – Devstringx

We are going to identify the graphics content of a mobile app using the Appium image recognition library.

We are using an image element (a base64-encoded image file) as a template to compare it with the image captured from a device screen.

Let’s see how this approach works.

Prerequisites

  • Appium
  • Java
  • Nodejs
  • Android Studio
  • IDE

Get Started

Install the latest version of Appium

npmi -g appium

Install OpenCV image comparison library

npmi -g opencv4nodejs

Detailed installation guide for OpenCV library here

Methods Used

  • Locate Images
publicMobileElementgetAllImage(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {

                        By img = By.xpath("//android.widget.Image");

                        waitForElementVisibility(driver, img);

                        MobileElementcurrentImg = null;

                        List<MobileElement>imgs = driver.findElements(img);

                        for (inti = 0; i<imgs.size(); i++) {

                                    if (imgCount == i) {

                                                currentImg = imgs.get(i);

                                    }

                        }

                        returncurrentImg;

            }
  • Capture Images From the App to Compare
public File doVisualCheckAllImages(AppiumDriver<MobileElement>driver, intimgCount) throws Exception {

                        String baselineFilename = VALIDATION_PATH + "/screenshots/app/img" + imgCount + ".png";

                        File baselineImg = new File(baselineFilename);

                       

                        File newBaseline = driver.getScreenshotAs(OutputType.FILE);

                        BufferedImagefullImg = ImageIO.read(newBaseline);

 

                        // Get the location of element on the pageto capture screenshot

                        Point point = getAllImage(driver, imgCount).getLocation();

 

                        // Get width and height of the element to capture screenshot

                        inteleWidth = getAllImage(driver, imgCount).getSize().getWidth();

                        inteleHeight = getAllImage(driver, imgCount).getSize().getHeight();

 

                        // Crop the entire page screenshot to get only element screenshot

                        BufferedImageeleScreenshot = fullImg.getSubimage(point.getX(), point.getY(), eleWidth, eleHeight);

                        ImageIO.write(eleScreenshot, "png", newBaseline);

 

                        FileUtils.copyFile(newBaseline, new File(baselineFilename));

 

                        returnbaselineImg;

            }
  • Compare Result
publicvoidcompareResult(AppiumDriver<MobileElement>driver, String checkName, String checkName2)throws Exception {

                        SimilarityMatchingOptionsopts = newSimilarityMatchingOptions();

                        opts.withEnabledVisualization();

                        File Img2 = newFile(VALIDATION_PATH + "/screenshots/app/" + checkName2 + ".png");

                        File Img1 = newFile(VALIDATION_PATH + "/screenshots/app/" + checkName + ".png");

                        try {

                                    SimilarityMatchingResultres = driver.getImagesSimilarity(Img1, Img2, opts);

                                    // If the similarity is not high enough, consider the check to be failed

                                    if (res.getScore() <MATCH_THRESHOLD) {

                                                File failViz = newFile(VALIDATION_PATH + "/FAIL_APP_" + checkName + ".png");

                                                res.storeVisualization(failViz);

                                                System.err.println(String.format(

                                                                    "Visual check of '%s' with '%s' failed; similarity match is only %f, and below the threshold of %f. Visualization written to %s.",

                                                                        checkName, checkName2, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));

                                               

                                    } else

                                                // Otherwise, it passed!

                                                System.out.println(String.format("Visual check of '%s' with '%s' passed; similarity match is %f",

                                                                        checkName, checkName2, res.getScore()));

                                   

                        } catch (

 

                        Exception e) {

            System.err.println("Visual check of " + checkName + " with " + checkName2+ " failed; Both images are expected to have the same size in order to calculate the similarity score.");

 

                        }

            }

Test Run:

publicvoidverifyPictures() throws Exception {

app.doVisualCheckAllImages(driver, 0);

app.doVisualCheckAllImages(driver, 1);

app.compareResult(driver, "img0", "img1");

}

Recommended To Read – Mobile App Automation Testing Using Appium and Python

Test run console

If you are interested in even more software testing-related articles and information from us here at Devstringx, then we have a lot to choose from for you.

FAQ

  1. How do I verify an image in Appium?

To verify an image in Appium, you can use Appium Image Recognition or Appium Screenshot Comparison. Appium Image Recognition is a feature that allows you to verify an image by comparing it with a reference image. You can use this feature to automate tests for scenarios where visual verification is required.

  1. What is Appium Inspector and why is it used?

Appium Inspector is a tool that use to inspect the user interface elements of an application. It provides a graphical representation of the application’s user interface, making it easier to identify and interact with specific elements.

  1. How do you click on an image in Appium?

To click on an image in Appium, you can use the Java Image Recognition library. This library allows you to locate an image within the application’s user interface and interact with it programmatically. You can use this library to find an image and then perform a click action on it using Appium’s API.

  1. How do I automate face recognition in Appium?

Automating face recognition in Appium can be challenging, as it requires specialized software and hardware. One way to approach this is to use a face recognition API, which can integrate with Appium. You can use this API to recognize faces in images or videos, and then perform automated actions based on the results.

  1. How do I find image recognition?

There are several tools and libraries available for image recognition, including Appium Image Recognition and Java Image Recognition. It allows you to verify images in Appium, while Java Image Recognition provides a library for interacting with images programmatically.

Share this post

Back to Blog