Using color algorithms in Automated Visual Testing

More complex UI issues

After static image processing proved to be useful in our everyday automated regression testing (read about it here), we decided to cover more complex UI issues using visual testing concepts. Many applications in our company contain dynamic (interactive) UI elements, which have to be checked manually each time new features are added.

The best way to picture such an example is to expand our previously explained case where the application contains map functionalities. Instead of using a static image as a map and map layers preview, we can use interactive maps, which are not static by any means. Such an example can become tricky to automate, as interactive maps are not static images (they can take time to load, and their components can have some offset in preview depending on service you are using). Even though you can play around scaling images or setting some specific time for taking screenshots, these tests will become flaky over time, and you will end up having too many special cases and complex code to maintain. Also, real nightmares start when you have multiple browsers, devices or headless mode to run your visual tests, as image comparison depends tremendously on screen resolution. Therefore we contemplated a new way to handle such situations. The optimal solution in our case was to combine functional and visual testing paradigms to avoid manual testing. The ultimate goal was to be sure at any time that maps, markers, and map layers are loaded. Other UI components linked to maps (modal titles, map description, or legends) could be automated using standard regression testing (via HTML elements check).

How to automate interactive visual elements?

We found using the dominant color algorithm, in combination with processing screenshots, as the most effective way to overcome issues with interactive elements.

Following flow proved to be successful for our test scope:

• Identify image area (canvas or another HTML element)
• Take screenshot of such area
• Use ImageMagick options to center and standardize image (to avoid frames and to minimize offset)
• Use quantize method to extract N most frequent colors (number depends on which feature is tested)
• Use the histogram to show the most frequent colors in decreasing order
• Save histogram and convert its colors to hexadecimal values
• Check if expected most dominant colors exist on screenshot.
• If the map has map layers (pins, markers, heat-map, or another data preview), base-map can be turned off, and layers existence checked. This can be done by checking if white is not the only dominant color for selected UI element (which would mean that element is empty – no layers added)

This method does not give information if maps are always loaded identically – which is the main criterion for static images. The valuable information we get from this method is if the map is loaded at all, and if its layers appear every time they are supposed to. This method saved a tremendous amount of time we spent manually checking if our map services work as expected (as some of our applications rely very much on maps and similar geo-services). Also, we found this method useful if the application contains some kind of ads (especially video ads), which helped us get away from the situations where our links were broken and we did not notice it on time.

Test implementation

Various implementations of a dominant color algorithm can be found over the Internet, depending on the programming language and testing framework used for visual tests. As we mentioned before, our tests are written using RSpec, Watir, and Ruby, so we found RMagick gem (a binding from Ruby to the ImageMagick image-manipulation library) to be most suitable for the test implementation. Hence, we found a dominant color algorithm implementation that uses RMagick.

Original map

Before we dive into algorithm details, we will show you how it is used for testing purposes, on a dummy map example. The image below shows a base-map with red circles showing the intensity of earthquakes in certain geographic areas. 

Those circles represent the map layer. For the purpose of showing dominant color algorithm usage, we set up the following scenario:

• The test should check if map layers are loaded correctly. In such a case we do not need to know if maps are generated below such layers, so we configured the test environment to show only layers in the selected HTML elements.
• This way we can focus on layers of interest only. The image can be found below.

            Image used for layer testing

All steps regarding test implementation will be explained in the following sections.

Identify an area of interest and save it as an image

Similar to the still image identification, we used Watir to identify where our maps are shown (we are fetching element location via Watir XPath wrapper). RMagick method from_blob() in combination with resize_to_fill() is used to manipulate saved image so that it is resized to fill certain resolution. Such a normalized image is saved in the desired folder.

context "Write image in png format to the path" do

  it "saves map as image" do

    #Decoded image is resized to fill certain resolution using Center Gravity    option

    image_data = Magick::Image.from_blob(Base64.decode64(image_got_from_canvas[‘data:image/png;base64,’.length .. -1]))

    img = image_data[0].resize_to_fill(1200, 600, gravity = ::Magick::CenterGravity)

    #Files saved in lib/data folder of test project

    path = “././lib/data/#{file_name}.png"

    img.write(path)

  end

end

Use image quantize method to extract N most dominant colors

After the image is saved, that image should be quantized. Quantize method is used to reduce image colors to N most frequent ones, by using the RMagick quantize() method. This is the first method used to implement the dominant color algorithm.

context "Extracts colors from map" do

  it "chooses N colors" do

    path = "/lib/data/#{file_name}.png"

    image.image_path =  Magick::Image.read(path)[0]

    #Extract N most frequent colors, e.g 5

    image.quantized = @homepage.quantize(comparem.image, 5)

  end

end

Create histogram from quantized image

Another method from the mentioned algorithm is used to manipulate the quantized image. Color histogram is created from such an image, and that histogram is sorted by decreasing frequency. color_histogram() method from RMagick is used.

it "creates color histogram" do

  #sorts colors by frequency and saves them as pixel preview

  image.sorted_histogram = @homepage.sort_by_decreasing_frequency(image.quantized)

  image.sorted_histogram.write("././lib/data/#{image.histogram_name}.png")

end

The histogram we created from our example can be seen here:

Convert histogram values to hexadecimal color notation

To be able to create an assertion for our tests, histogram colors are converted into hexadecimal values. Those values are created using the to_color() method which converts any pixel color to adequate colors using All Compliances.

it "gets image colors" do

    #gets colors included in histogram as hexadecimal values

    image.color = @homepage.get_pix(image.sorted_histogram)

  end

end

Create assertions for checking selected UI elements

Finally we can check if our color histogram shows any other colors then white (#000000 in hexadecimal notation). If this is not the case, that means that our layers are not loaded, and the test will fail.

context "Checks if map has layers" do

  it "map has layers" do

    #If map contains only white color (hexa value: #000000), it is blank, so layers do not exist

    if image.color.size > 1

      image.color.each do |color|

        #color is converted from RGB coeficients to hexadecimal value by to_color(Magick::AllCompliance, false, 8, true) method

        expect(color.to_color(Magick::AllCompliance, false, 8, true)).not_to be_eql "#000000"

      end

    else

      expect(image.color[0].to_color(Magick::AllCompliance, false, 8, true)).not_to be_eql "#000000"

    end

  end

end

Output of the test:

Write image in png format to the path

  saves map as image

Extracts colors from map

  chooses 5 colors

  creates color histogram

  gets image colors

    #FFFFFF

    #FECFD1

    #FC8F93

    #D8D5D6

    #8D7F80

Checks if map has layers

  map has layers

Created histogram (saved as image):

Algorithm implementation

In previous steps, we described how the dominant color algorithm is used in our scripts. Now we will show algorithm methods we used for testing (complete algorithm can be found here):

#Methods for getting image histogram and colors present in the picture

  # Histogram is created as 1-row image that has a column for every color in the quantized

  # image. The columns are sorted decreasing frequency of appearance in the

  # quantized image.

  def sort_by_decreasing_frequency(img)

    hist = img.color_histogram

    # sort by decreasing frequency

    sorted = hist.keys.sort_by {|p| -hist[p]}

    new_img = Magick::Image.new(hist.size, 1)

    new_img.store_pixels(0, 0, hist.size, 1, sorted)

  end

  def get_pix(img)

    a=[]

    pixels = img.get_pixels(0, 0, img.columns, 1)

    pixels.each do |p|

      puts p.to_color(Magick::AllCompliance, false, 8, true)

      a.push(p.to_color(Magick::AllCompliance, false, 8, true))

    end

  end

  def quantize (img,num)

    # reduce number of colors

    img.quantize(num, Magick::RGBColorspace)

  end

When to use algorithms in visual testing?

Diving deep into image processing algorithms, to be able to use them in visual testing, can be seen as to much work and to little value for QA team. This is true in case you have few images to check, or simple application to test. But when you have complex business logic or complex application, where there are intense sets of rules for image presentation and appearance, this kind of tests can be of great help and also time-saving. In our case it was worth the effort as we:

• do not spend our time comparing images manually anymore
• do not have to analyze complex business rules to see if images are generated correctly

Having all this in mind, it is up to the QA team to decide if they are willing to dive into such adventure and if the output is good enough to replace manual testing.

Conclusions

In our blogs regarding UI testing, we explained two methods that proved the most efficient for our automated visual testing. Those methods can be further expanded to test whole pages. Also they can be combined to get more precise and more stable tests. Our future goal is to create a whole visual test suite that will be reliable as much as our standard regression tests are. Previously explained algorithms set us on a good path to achieve these goals.