Testing an application user interface solely by human quality assurance engineers is no longer feasible because of human error risks and the extraordinary complexity involved. That’s why test automation is now standard among most software development and testing companies – according to QA consultants from A1QA. In this regard, machine learning in conjunction with computer vision (feature detection technologies like KLT and KAZE) already contributes to automated testing significantly.

In recent years, developers used test recorders like Selenium to play back a user’s gestures on a UI following a code change. Limitations of this type of test automation inspired an immense community of developers to contribute enhancements to Selenium, and by far the most intense focus is to add computer vision ML algorithms to the testing and verification of web app UIs.

Remote surgery systems such as Da Vinci require computer vision ML-aided testing too. This system uses a computer vision ML recommender to advise doctors by making suggestions during an operation. Da Vinci represents a growing class of medical procedure assistance software which requires absolute testing validation because of the high risk associated with surgical procedures.

It’s possible to take a backdoor view into proprietary enterprise deployment of computer vision ML through typical job descriptions for the position of QA engineer. For example, Tesla which develops an autonomous vehicle UI regularly seeks engineers to develop computer vision ML for diagnostics. Another popular car manufacturer farms out test automation of its embedded dashboard UI to a San Francisco company which uses machine learning and computer vision to verify that the UI complies with design specifications. In this case, a critical design specification is performance under load that calls for functional testing.

The Testing Models in Detail

An essential requirement of functional testing is to determine how the app performs when hundreds of users simultaneously access it and send requests including database queries, account creation, and UI dropdown views populated by conditionals. Even dedicated testware companies don’t have enough workstations in-house to run this level of testing, so how do they do it?

An increasingly popular model is the virtualized, or containerized application. The way this works is that a container system like Docker builds a virtual model of the required OS to host an instance of both an OS and the app under test. In this setup, developers configure a browser like Chrome to open the app inside the virtual machine and execute gestures on the app’s UI as would an actual human visitor to the web app. The virtualized app includes a display emulation such as Google’s display widget, which can simulate thousands of devices and display types.

As prescribed by a test script, the testware operating in conjunction with the browser captures a “screenshot” at a critical juncture, even though there is no actual “screen,” and then records this for inspection by a computer vision ML algorithm. If the algorithm decides that there is an anomaly, it then presents the screenshot in an automated report, usually created in HTML format for the human tester to evaluate.

The errors which computer vision ML algorithms are good at detecting now include:

• Page element relocation
• Page rendering errors
• Visual completion
• Benchmarking of visual components
• Menu and text accuracy with OCR
• Calculated value accuracy

Which ML methods are actually used to detect such errors in a simulated virtual space? Developers use an ML method called convolutional neural networks (CNN) to identify page transitions and determine rendering accuracy and completion, and in particular to verify compliance with design specifications. The BlackMagic tool can capture screenshots to be evaluated by a CNN algorithm for accuracy, yet this evaluation is complicated by page transitions which are different all the time since they come up uniquely from a recommender system, such as YouTube’s suggestions, for example.

Image classification can be accomplished by Inception-v3 classifier, which is integral to the TensorFlow ML library. In the same Stanford research cited above, 90% confidence levels of accurate image classification were achieved in testing a TV UI with YouTube page transitions. Non-CNN methods including clustering algorithms such as Bayer-Moore majority vote and K Means achieved the best accuracy in this UI testing, which showed almost flawless classification on diverse and distinctive training datasets. It’s important though that these algorithms must be deployable in cloud-based UI testing orchestrations.

In cloud-based automation testing, everything is virtual, and human error often explodes at an exponential scale. In order to simulate a real user, an automated UI tester must go through 3-stage authentication, that is, to initiate login, check the email, and then enter the dynamic authentication code in the app.

This stage of test automation does not need machine intelligence but it does require a bit of human attention. Developers working in this automation space face a familiar human weakness: forgetfulness. A common pitfall in configuring a test orchestration is to leave a virtualization running out of control – meaning to spin up an orchestration of virtual users that doesn’t know how to turn itself off. An error of this kind can be costly, especially when testware companies charge up to $3,000 per month, and virtualizations charge per nanosecond of CPU time!

Automated UI app tests can also effectively identify the relocation of a page element when a developer makes a code change. A user input field might change from green to sea green and relocate across the page, for example. Computer vision ML using feature detection algorithms like KLT can track this change and update a test script so that it continues to function across integration cycles without the need for editing by a QA engineer.

Conclusion

This advance in automation testing is a substantial relief to QA departments and helps to keep up testing with other stages of continuous deployment. The intense and widespread development guarantees that the evolution of computer vision ML-based automated UI testing will keep maturing and become a standard practice for all competitive software development enterprises.

 

About the Author: Having more than 8 years of writing experience, Smridhi Malhotra is a professional tech, health and travel blogger.  She loves to gather and share her profound knowledge about latest developments in technology. Smridhi is a management graduate and visual graphics artist and is currently pursuing masters in behavioral psychology. Her hobbies are practicing mindfulness, counseling children and traveling (a special love for Africa).