80% Reduction in Testing Efforts for A VR Gaming Platform from Innominds’ Test Automation Labs

Game testing is a specialized skill that requires diverse tests to be performed in order to meet complex architectural needs and recent emergent compliance rules in game development. The number of elements interacting with each other inside a game is large, so identifying and testing them takes up a large chunk of the time and effort of test engineers. 

In 2020, smartphone games accounted for almost 50 percent of video gaming revenue worldwide. As mobile penetration rates and smartphone usage continue to accelerate on a global scale, mobile games revenue is on track to surpass the 100-billion-dollar mark by 2023.  

Such widespread game enthusiasm across devices and platforms makes game testing an important area of focus for the makers. Besides, fast performance is critical to gamers, being cited as the most important aspect of gameplay across all age groups, according to the State of Online Gaming 2020 research report from Limelight. Games that perform better in terms of the gameplay and stability of upgrades subsequently gain more popularity in the gaming world. 

When one of the most popular game manufacturers decided to up their game testing skills and test framework, they associated with Innominds to perform tests and establish the Quality Engineering (QE) standards for their game applications. They wanted to reduce the time and effort that goes into the extensive processes of functional, compatibility, and compliance tests by bringing industry-grade tests to their innovation labs.  

The game manufacturer is a consumer electronics company that develops the world’s leading VR Platform with supporting applications to the end-users. They specialize in developing immersive VR content/games using Unity and Unreal game engines. 

The new VR game app under testing was built on a Chromium Embedded Framework (CEF) and based on the WPF desktop UI platform.   

The existing game QA and release were becoming more and more challenging for their engineering teams. Therefore, Innominds assisted the client in setting up a QE automation lab and provided solutions to overcome common nagging issues. 

Innominds was able to mitigate the client’s concerns in the end by leveraging advanced tools and frameworks. We automated the testing process and reduced the testing effort by almost 80%.  

But what makes game testing different from regular app testing? The complexities in the number of elements used in a game force a different Quality Engineering (QE) methodology altogether.  

Read more: Car Sales Solution Finds New Trajectories in Quality Engineering with DevOps and Test Automation 

Innominds executes video game testing with test automation that developed 500+ test scripts 

 
The United States alone harbors much of the gaming interest of the world. But in recent years, other countries started to gain momentum, making game development a global enterprise. Owing to such popularity, competition is on the rise. To maintain a steady revenue, companies must put out quality products in the market to stay relevant. Testing is subsequently a driver for harnessing the gaming app market revenues.  

Game testing can be categorized mainly into two types based on the overall goals of testing i.e., black box and white box testing.  

1. Black box testing: In this type of testing, the QA team mimics the behavior of the average user. It tests for the game’s functionality as opposed to the inner workings of the code or database. 
   Black box testing also deals with the Graphical User Interface (GUI), load performance, and visual appeals, such as the special effects and animations. It focuses on the controls, features, gameplay, buttons, sounds, the passing of the levels, etc. In short, it analyses whether the game is playable, as a form of beta testing before it the games are released to the public. 
   Test automation scripts are generally built around gameplay to track progress and faulty controls or states. Image recognition is an AI sub-field that can assist any image-based testing. 
    
2. White box testing: This type of testing requires complete architectural and system knowledge. It includes database functions, libraries, game architecture and engine, third-party components, external players, etc. 
   White box testing is usually performed with unit tests, integration tests, and system compatibility tests. 

Testers use black box and white box forms of testing to majorly allocate tests and identify the major client challenges in testing.  

The Quality Engineering (QE) team at Innominds helped the client in areas of challenges through automated solutions which brought significant results. This came with over 500 test scripts that were developed to guarantee faster QE cycles.  

These challenges were mainly in these three areas of testing: locating UI elements and images, parallel tests across different platforms, and customizing the automated test reports.  

1. Difficulty in locating UI elements and images- UI testing  

The client faced difficulty in locating UI elements and images for UI testing. GUI (Graphical User Interface) testing is generally used by testers to validate the UI elements and also to identify any defects in them.  

But when the regular object spy fails to capture the elements, additional framework implementations may be needed to make things work. Innominds designed and developed an automated testing tool that can identify elements by using existing object spy. It delivered the additional UI automation support. 

2. Parallel tests across platforms and devices- Compatibility testing

One of the most challenging aspects of game testing is testing for compatibility across various devices, browsers, and operating systems. The client experienced difficulty in compatibility testing for so many different versions of OS in combination with devices. 

Game developers are usually concerned with the number of tests to run to ensure that such a wide range of user bases get full access to all the functionalities. If the games are tested on mobile devices, the complexity increases further as iOS and Android versions get upgraded regularly, but not necessarily uniformly, for all new device releases.  

Innominds set up a state-of-the-art test lab to solve the cross-device, cross-platform testing issues with around 50 system configurations. The configurations would test for all hardware and OS combinations.  

As a result of these tests, we were able to make early detections of defects that were specific to platforms and browsers.   

3. Customizing automated test reports - Test coverage and report generation

Ideally, after the tests are performed, generating a test report is equally important to further the testing efforts in the future. Such reports analyze the test results to make decisions based on them.  

Test reports gather the summary of test activities and assess the success of each test. Depending on the quality of the results delivered, the software releases can be planned and executed.  

Since the client’s test coverage suffered, Innominds helped in developing tests scripts that covered all the major business scenarios in terms of compatibility, such as different integrations with different web and desktop components.  

Our solutions included: 

• Customized test reports generated on different machines 
• Development of a consolidated test report after parallel test executions on multiple machines 
• Development of test scripts covering all the major business scenarios, UI validations, and integrations between various web and desktop components 

Quality engineering for games is meant to ensure that the optimal quality is maintained with each release. As global game regulations become more exacting, game developer companies need to find more innovative methods to fasten up the testing process while still maintaining the test accuracy.  

It can only be assumed that gaming is going to gain more popularity through different channels, as most of the app store purchases in iOS and Android are accounted to game purchases. Thus, games are the most hot-selling commodity in the mobile world.  

To know more about the methodologies employed by Innominds QE for game development that helped the client achieve an 80% reduction in testing efforts, check out the case study.