AI to Make Visual Testing Smarter – InApps Technology 2022

A new phone in a new size might delight consumers, but it’s a headache for organizations trying to standardize the look of their applications across devices.

“Companies spend a lot of money building beautiful apps, but they also have to make sure those apps function correctly on all the different devices and browsers,” said Adam Carmi, chief technology officer and co-founder of the visual testing startup Applitools.

It’s focused on automating testing the way the application looks, as the human eye would see it. Actually, its algorithms aim to do the human eye one better, to be pixel perfect with technology it calls Application Visual Management (AVM).

AVM combines testing, image processing and artificial intelligence tools to identify when applications look visually incorrect when released or when a platform, device or browser changes.

“The UI is extremely sensitive to any difference in the device, the browser, the platform. Every pixel can cause the button to overflow and get out of the screen. Every change a developer does in CSS can change other places in the application, causing problems with the brand, disabling the application completely, end up costing a lot of money because the app can’t be used,” Carmi said.

The company’s most recent release is called UltraFast Visual Grid, which allows users to specify a combination of devices, browsers and view port sizes to test all at once. Its SDKs upload Document Object Model (DOM) snapshots — not screenshots — and containers so they load faster and more reliably.

“We have special SDKs that rather than grabbing an image, they grab all the resources from the page, the CSS, the HTML, the images, the fonts, upload these to our grid that has its own devices and browsers, then render the screenshots across all of them in parallel. This means you are running all your tests locally, even on a dev machine.

“A developer could be sitting in a coffee shop on his Mac machine doing 20 tests covering everything in his app on different mobile devices, on a Windows machine running IE 10, without him even needing such an environment and without waiting 20 times to run all these tests. It would be much faster and cost much less than setting up a real device lab,” Carmi said.

Cross-browser and cross-device testing are not cost efficient. On the client side, 99 percent of the bugs are not client-specific, he says.

“We argue that you can write a single test that covers these edge cases, run this test on 20 devices, but you have 1,000 other tests you need to run without all this overhead. There’s no point in running all these tests 20 times. You can do things in a much smarter way, save a lot of money and get much more out of them.”

It also makes use of the Visual Grid supports Selenium Java, Selenium Javascript, Selenium C#, Selenium Python, Selenium Ruby, Webdriver IO, Cypress, Storybook, as well as the Selenium IDE extension for manual QA and marketing teams. The company plans to support 50 SDKs by the end of the year.

The company also has assumed leadership with Selenium IDE, an open source tool to record and playback test automation for the web. It originally was a browser extension in Firefox, but the APIs were no longer supported.

It enables QA engineers who cannot code to write and maintain automated tests.

The new Selenium IDE is designed as a modern browser extension with support for Chrome and Firefox, with support for other browers in the works. The company has created an integration between the IDE and Applitools Eyes, its AI-powered visual testing and monitoring solution.

“We took Selenium IDE and rewrote it from scratch. Now it’s a powerful, modern tool. It can play back those tests from your continuous integration servers, and we’ve added an extension that allows you to record visual tests,” Carmi said.

Difficult Problem

In 2012, Carmi was running a R&D organization at an Israeli-based security company.

“We white-labeled the UI for several brands and localized it in six different languages. We had a white label for Fujitsu. These guys were so strict about the UI. In 2012, the UI was not so important then. But these guys were meticulous — the letter is a pixel to the side, the color is not exactly the proper shade, titles in Japanese, of course, are different… We were just unable to meet their quality demands,” he said. “It would take us an entire week to test everything and still we would have UI issues.”

He started searching for a tool with vendors such as IBM, Microsoft, HP, to no avail. Functional testing, yes; visual testing, no. So he started building one himself. He found it incredibly difficult.

“It took me eight months to get something that was barely working. Moshe [Milman], my co-founder, loved it and we founded the company in 2013,” he said.

“By the time we got a very solid tool, the world transformed … and suddenly the UI was the most important piece of every application.”

The company now is based in San Mateo, California, with R&D offices in Tel Aviv. Its customers include Sony, Mastercard, Salesforce, Gannett and the Israeli government, which is constantly under attack from hackers trying to deface its websites, he said.

There are about 30 open source projects for visual testing such as PhantomCSS, Huxley, Wraith, Shoov, Gemini. Commercial offerings include the likes of Screener.io and Percy.io. Carmi considers them all toys, without the maturity or functionality of Applitools.

Applitools, he said, for instance, can ignore changes in spacing when the content didn’t change. You can ask the tool not show you a header change that will pop up on every test across different environments. It can handle recurring patterns, say a retail site that one day features seven products, but the next day only four. If there’s an ad that’s constantly changing, you can mask it so it will be ignored. If something is moving, you can designate a floating region in which that movement is OK.

“Similar to the open tools, [the commercial offerings] are a little bit better, but they have nothing around AI for maintenance, dynamic data in layout, cross-device, cross-browser testing, support for different SDKs. We support over 40 SDKs, all the main test technologies for web and mobile. If you want to test PDFs, we support that. If you have a set-top box or IoT device that can’t be tested in a standard way, if you get a screenshot, we can validate it,” he said.

In addition to using to monitor a website, localization, and security, as in the Israeli government example, it also can be used for root cause analysis. It doesn’t just show the changes, but what in the implementation of the page caused those changes, he said.

Feature Image: “Devices” by Jeremy Keith. Licensed under CC BY-SA 2.0.