Ensuring a flawless UI is crucial for customer retention and loyalty. For example, Stanford Study shows that over 90% of people say that web design affects their first impression of a site, and over 70% of people will base a company’s credibility on visual elements.

In automated tests, ensuring the functionality and stability of an application is most important. However, most automated tests often fall short when detecting visual defects and ensuring a seamless user experience.

What is Visual Testing?

While functional testing aims to verify that the application’s features and functionalities are working as intended, a good addition to this kind of testing is visual testing, which verifies that the overall layout of the application and the application’s visual elements are displayed properly and consistently across various devices, operating systems and browsers. By using this approach, automated tests can catch even the slightest visual anomalies that are sometimes harder to catch even with manual testing, ensure pixel-perfect precision, and enhance user satisfaction.

Why Use Visual Testing in Automation?

This blog explores some of the key benefits of visual testing in automation:

Regression Testing: Ensuring that changes or updates to your application haven’t introduced any unintended visual defects. Visual testing can quickly identify any visual inconsistencies by comparing before and after screenshots, thereby accelerating bug fixing.

Improved User Experience: Ensures application’s user interface (UI) is presented as intended, providing users with a visually pleasing and intuitive experience. Detecting visual defects such as misaligned elements, broken layouts, or incorrect color schemes can prevent users from encountering frustrating UI issues.

Comprehensive Testing: While functional testing focuses on an application’s underlying logic and behavior, visual testing completes it by verifying the visual aspects. It covers a wide range of visual elements such as fonts, images, icons, spacing, and responsiveness across various devices and browsers. This approach ensures that the application looks and works as expected for all users, broadens the test coverage, and provides more confidence in the overall quality of the software.

Seamless Integration with Existing Testing Frameworks: Visual testing tools can often be seamlessly integrated with popular testing frameworks and continuous integration/continuous delivery (CI/CD) pipelines. This makes it easy to incorporate visual tests into your existing test suites and automation workflows without requiring major changes to your testing infrastructure.

Without visual testing being implemented in automation, visual issues might be present without breaking the application’s functionalities, whereas automation tests would pass even though the issue in UI is present.  One example of this kind of issue can be found in the image below.

Implementing WebdriverIO Image Comparison

Many automation frameworks support image comparison but, for the purposes of this blog, we will be using WebdriverIO (WDIO). WDIO is an open-source testing framework for automating web applications. It provides a simple and flexible way to write and execute end-to-end tests, making it a valuable tool for ensuring the quality and reliability of web applications. WDIO has a very useful service that can be imported into automated tests to fulfill the role of visual testing. Image comparison in WDIO allows you to compare screenshots to identify any visual differences between them. This can be useful for visual regression testing, where you want to ensure that the UI of your web application remains consistent across different versions or updates.

WDIO Image Comparison uses pixel-by-pixel comparison, which means that each pixel is being compared from the baseline image with the resulting image, giving visual testing a pixel-perfect precision.

To perform image comparison in WDIO, you can use the webdriver-image-comparison package, a plugin specifically designed for this purpose. Here’s a step-by-step guide on how to set it up and use it:

Install the package:

npm install --save-dev wdio-image-comparison

Configure WDIO to use the image comparison plugin. In your WDIO configuration file (usually named wdio.conf.js), add the following code:

const { join } = require('path')

// wdio.conf.js
exports.config = {
    // ...
    // =====
    // Setup
    // =====
    services: [
        [
            'image-comparison',
            // The options
            {
                // getBaseLineFolder() and getResultsFolder() are methods that save baseline and resulting images to corresponding     folders. More details about these methods in subsequent chapters.
                baselineFolder: getBaseLineFolder(),
                screenshotPath: getResultsFolder(),
               //Save the images per instance in a separate folder so for example all Chrome screenshots will be saved in a chrome    folder like desktop_chrome. We won’t use this because we are creating our own folders for baseline images so it is set to false.                       savePerInstance: false,
//Saves baseline image automatically after first run without manual interaction
                autoSaveBaseline: true,
//Automatically block out the status and address bar during comparisons. This prevents failures on time, wifi or battery status.
                blockOutStatusBar: true,
//Automatically block out the tool bar
                blockOutToolBar: true,
                // NOTE: When you are testing a hybrid app please use this setting. A hybrid app is one that can be downloaded and installed on multiple mobile platforms like Android and iOS
                isHybridApp: true,
                // ... more options
            },
        ],
    ],
}

WDIO Image Comparison has many more plugin options that can be set in configuration or test scripts based on needs and type of plugin. More about this can be found in references. 

Perform image comparison using the checkScreen()  or checkElement() methods. The checkScreen() method is used to capture a screenshot of the entire page, whereas the checkElement() method is used when capturing a screenshot of a specific element on the page. These methods compare the current screenshot with the reference screenshot and return a result indicating whether they match. Assertions can be used to check the result. Since WDIO image comparison uses pixel-by-pixel precision, a tolerance can be added in these methods. Tolerance can be used if some difference in the screenshots is expected and we want to ignore it. An example of when the tolerance could be applied will be provided in a subsequent chapter.

With the above steps, you can use WDIO’s image comparison capabilities to detect visual differences in your web application’s UI during automated testing. Remember to update the file paths and options according to your project’s structure and requirements.

Using WDIO Image Comparison Service

Example:

  • We will create three test cases
    • The first test case will capture a screenshot of the entire page
    • The second will capture an element of the page
    • The third test case will capture a screenshot of the entire page with tolerance
  • We will run these tests on different resolutions to cover a variety of devices
const HomePage = require('../pageObjects/HomePage'),

homePage = new HomePage()

describe('Visual testing with WDIO image comparison', () => {
    it('Verify landing page is displayed as expected', async () => {
        await expect(
            await browser.checkScreen('landingPage',
                {
                    /* some options */
                })
        ).toEqual(0)
    })

    it('Verify one element on page is displayed as expected', async () => {
        await expect(
            await browser.checkElement(
            await homePage.navigationBar,
            'elementOnPage',
                 {
                    /* some options */
                 }
            )
        ).toEqual(0)
    })

    it('Verify landing page is displayed as expected with given tolerance', async () => {
        await expect(
            await browser.checkScreen('landingPageWithTolerance',
               {
                    saveAboveTolerance: 1.0
               })
        ).toBeLessThanOrEqual(1.0)
     })
})

In the first test, we are using the checkScreen() method that captures the entire screen and makes a baseline image, and every next run compares a new screenshot with the baseline screenshot. Comparison is done on the name of the screenshot. Examples of baseline images on different devices:

Desktopvisual testing

iPhonevisual testing

iPad
visual testing

iPad Landscape

visual testing
In the second test, we use the checkElement() method, which captures a screenshot of elements of the page that can be compared in the same way. 

Desktop

 

iPhone

 

 

 

iPad

 

 

iPad Landscape

 

 

When some inconsistencies are captured, they will be saved in the resulting folder, and the difference will be marked with a color.

visual testing
When a mismatch is detected, the test will fail.


Advanced Usage of Image Comparison Methods

In the previous examples, methods for capturing screenshots of a page or element were invoked multiple times for several cases. Suppose one test needs to capture screenshots of different pages or elements several times. In that case, this implementation can quickly become very hard to maintain since we will end up with many hardcoded values and repeating code. A good approach to solving this issue can be to create a generic method that will be declared in the helper file and called on demand. An example of this method can be found below:

async function captureScreen(screenshotCaptureObjects = []) {
    for (const screenshotCaptureObject of screenshotCaptureObjects) {
         expect(
             await browser.checkScreen(screenshotCaptureObject.screenshotName, 
 	 	{
 	 	     saveAboveTolerance: screenshotCaptureObject.tolerance,
 	 	})
 	 ).toBeLessThanOrEqual(screenshotCaptureObject.tolerance)
 	 if (screenshotCaptureObject.element) {
 	     await screenshotCaptureObject.element.scrollIntoView()
 	 }
    }
}

This method (captureScreen) accepts as a parameter an array of objects that will be declared on the script level. An example and explanation of this parameter will be explained later. When the array of objects is passed as a parameter, the for loop will iterate through every object and property in the object and take screenshots based on how many properties are declared. Two properties are mandatory to be declared: screenshotName and tolerance, but if we don’t want to use tolerance, we can set it to zero. Element is an optional property. We can declare this property when a scroll to a specific element is needed. This is useful when working with the checkScreen() method and smaller resolutions. For example, let’s say we need to capture three screens on mobile resolution for the same page. This method will iterate three times and scroll to element after each iteration to “prepare screen” for capturing a screenshot for the next iteration. Method captureScreen() can be modified to use the checkElement() method, but for the purpose of this blog, we will use this variation.

As mentioned, an array of objects needs to be declared on script level as a parameter to be passed to captureScreen() method. An example of this object is presented below:

const screenshotCaptureProperties = {
    landingPage: [
        {
             tolerance: 0.0,
             screenshotName: ‘landingPage’
        }
    ],
    landingPageTolerance: [
        {
             tolerance: 1.0,
             screenshotName: ‘landingPageTolerance’
        }
    ],
    landingPageScroll: [
        {
             tolerance: 0.0,
             screenshotName: ‘landingPageTop’,
             element: page.whyUsSection    
        },
        {
             tolerance: 0.0,
             screenshotName: ‘landingPageBottom’
        },
    ],
}

Since tolerance and screenshotName are mandatory properties, we need to declare them for each object. Element is optional, so we will declare it only when scrolling to an element is needed. After scrolling, another screenshot will be captured.

When the array of an object is defined, we can use the method like this:

await captureScreen(screenshotCaptureObjects.landingPageScroll)

When this method is called as presented, the landingPageScroll property will go through iteration in a for loop, and ‘landingPageTop’ will be passed as screenshotName, tolerance will be zero, and after taking a screenshot, scrolling to a defined element will happen after which another screenshot will be captured.

Challenges

Working with visual testing introduces several challenges that users may encounter. These challenges could include:

Timing for capturing screenshots

When working with a dynamic application, capturing a screen precisely at a specific moment can be challenging due to factors like prolonged page loading, concurrent element loading, and pop-up appearances. In such instances, false negative results may occur if the timing of capturing screenshots is imprecise, requiring repeated test runs to confirm the absence of actual visual differences. To ensure accurate comparison, we can utilize various built-in wait mechanisms that WebdriverIO offers before capturing the screen and doing comparison with the baseline image. Depending on what we are trying to capture, we can implement a wait mechanism on a specific web element or, for example, wait for a complete page to be loaded in various ways (wait for a steady page, wait for dom to be loaded, etc.)

Difference between consecutive run

In automation tests, we might use visual testing on pages containing data that might change with each run, for example, a unique ID generated and displayed on the UI or dynamic visual components that result in screenshots capturing different parts of the element during consecutive runs. If the difference is expected each time the test is executed, a tolerance in tests can be used to ‘allow’ some percentage of difference to be ignored. Since the difference is expected to be present each time, we can consider it to be a constant, and a tolerance won’t affect our test results because each time additional difference occurs, the percentage of difference will be higher, and the test will fail indicating that there is additional mismatch in comparison. Additional options are plugin options offered by the wdio image comparison service and can be added to improve visual testing in automation.

Runs on CI/CD environments vs local runs

In image comparison, baseline images are required to compare screenshots from each run. This can be challenging in CI/CD environments like Jenkins, where tests run on different machines with varying screen sizes and resolutions. In such cases, a separate set of baseline images should be created for CI/CD pipeline runs, a topic that will be explained in more detail later. An example of this case might look like this:Even though there is a slight difference between these two screenshots, automation tests will fail since this is a pixel-by-pixel comparison. This is the reason why baseline images should be saved in separate folders, and comparison should be done with corresponding baseline images.

Different screen sizes on different machines (integration with CI/CD pipeline)

As mentioned in the previous challenge, when integrating visual testing with CI/CD pipelines, we might face a challenge where tests run on different machines with varying screen sizes and resolutions. Although saving baseline images specifically for CI/CD tools like Jenkins is a solution, variations in screen size on different machines may still cause test failures. For example, one test can be triggered on several different machines with different screen sizes over some time and produce different resulting screenshots. This issue can be resolved by setting a specific resolution across both local and CI runs instead of using browser.maximizeWindow() method. It is important that the specific resolution set must be less or equal to the actual display resolution of the machine where the test will be executed, otherwise, it will be scaled down to the display resolution of the target machine.

Comparison with different baseline images

Since this blog covers visual testing on a variety of different resolutions (different devices), baseline images should be saved to according paths/folders. For example, suppose the test is executed on iPhone resolution. In that case, a corresponding baseline image will be created. Still, if the next test is executed on iPad resolution, it will be compared to the previously saved baseline image for iPhone resolution, and the test will fail. The solution for this case should include a helper file that will arrange baseline images to different folders. With each run, the current screenshot will be compared to the corresponding baseline image from the defined path. Example of helper file:

const getSpecificDevice = () => {
 process.env.RESOLUTIONBREAKPOINT = getResolutionBreakPoint()
 let device
 if (process.env.RESOLUTIONBREAKPOINT == 2) device = "iPadLandscape"
 if (process.env.RESOLUTIONBREAKPOINT == 3) device = "iPad"
 if (process.env.RESOLUTIONBREAKPOINT == 4) device = "iPhone"
 if (process.env.RESOLUTIONBREAKPOINT == 1) device = "Desktop"
 return device
}

const getBaseLineFolder = () => {
    if (process.env.JENKINS_RUN === 'true') {
        return join(process.cwd(),`./data/baselineImages/jenkins/${getSpecificDevice()}`)
    } else {
        return join(process.cwd(),`./data/baselineImages/local/${getSpecificDevice()}`)
    }
}

const getResultsFolder = () => {
    if (process.env.JENKINS_RUN === 'true') {
        return join(process.cwd(), `./data/resultImages/jenkins/${getSpecificDevice()}`)
    } else {
        return join(process.cwd(), `./data/resultImages/local/${getSpecificDevice()}`)
    }
}

Where RESOLUTIONBREAKPOINT is an environment variable also defined in globals.js as resolutionBreakPoint.

getBaseLineFolder() and getResultsFolder() are used in the wdio conf file in image comparison service instead of predefined paths for saving baseline images.

Image Rendering

When using visual testing for web pages that contain images, we might experience issues with rendering. If the screenshot of the page should be captured at the precise moment and the image is not fully rendered, tests will fail, indicating a difference in some pixels. This might not be clear immediately while examining the test results, but if we look at the image in more detail, we can see that sometimes pixelation happens on some parts of images. For the solution, we can consider adding an additional option ignoreAntiAlliasing. This option can be added in the configuration of the image comparison service in the wdio config file or directly in the assertion with checkScreen() or checkElement() methods based on the needs. If this happens only in some specific cases, we can consider adding this option in the assertion. In contrast, if this issue happens across several pages, a better option would be to add it in the service configuration.

Conclusion

In conclusion, visual testing is a valuable addition to your automated tests, enabling you to validate visual integrity along with functionalities. By implementing visual testing techniques and tools into the testing process, you can enhance user experience, catch visual defects early, and achieve a high level of visual precision in your applications.

Github repo:

https://github.com/ATLANTBH/testing-research/tree/master/visual_testing_wdio

References:

https://webdriver.io/docs/wdio-image-comparison-service/

https://github.com/wswebcreation/webdriver-image-comparison/blob/main/docs/OPTIONS.md

https://credibility.stanford.edu/guidelines/index.html#chi01a

https://www.browserstack.com/guide/visual-testing-beginners-guide

 

If you found this useful, check out other Atlantbh blogs!

oban
Software DevelopmentTech Bites
February 23, 2024

Background Jobs in Elixir – Oban

When and why do we need background jobs? Nowadays, background job processing is indispensable in the world of web development. The need for background jobs stems from the fact that synchronous execution of time-consuming and resource-intensive tasks would heavily impact an application's  performance and user experience.  Even though Elixir is…

Want to discuss this in relation to your project? Get in touch:

Leave a Reply