We collaborate with production schedules to coordinate our test plans, testing expected game deliverables each milestone, sometimes creating a breakdown of a game into up to 20 testable areas. Testable areas are assigned to Software Test Engineers (STEs) to plan coverage and resources needed to test. Part of the plan will determines when to use test automation, how much telemetry to track. We adapt our test plans in concert with production needs, and use bug and coverage metrics to redirect testing until it’s time to “shut down” a game for release.
All requirements are met as listed in the design document. The requirements will be tested individually using test scenarios. Functional testing will include testing areas such as menu navigation, game-play elements, AI, performance, play balance, save system, and other validation activities.
Compliance with certification requirements requires special expertise which we develop in our team members. Documented frequent exceptions and common failure errors help us to get first time passes. We avoid common mistakes of test groups that are not exposed to submitting hundreds of titles through a platform certification or compliance test group.
We verify that the current build meets the Production acceptance criteria for each milestone. A series of basic tests are executed (automated if possible) to determine whether gameplay, usability, design, and performance requirements for the milestone are met.
Our diverse teams create better testing coverage through thinking differently, together. We allow a testers to come up with their own ways to break the game in exploratory ways, testing within known limitations of partially-developed games. Directed ad hoc testing is done in a way that ensures coverage of game play features and desired areas to test (e.g testing southpaw controls in an area that has unusual control inputs). Usually this method is paired with structured testing to gain wider test coverage.
After an issue has returned from Production or Development, Test verifies the route they wanted to take with it. We confirm that fixes address original problems reported and that game features are implemented and working as intended. Testers also determine that changes haven’t broken previously released features. If the developer provides release notes with the builds, this can also be a helpful tool in the focus of regression tests. Regression testing is typically done as soon as a new build has been transferred.
Telemetry and analytical tools confirm and assist gameplay coverage for characters, maps, balance testing, controls, language and many other areas, and help us guide ongoing testing to the right areas. Tools can be used to automate play-throughs, speed up build verification tests, discover new bugs, and reduce the need for additional manual testing.
We coordinate our tech and staffing to meet the maximum capacity and testing scope for NxN-player games. Leveraging our latest technology to simulate extra players, running bug bashes with members of other teams that join for multiplayer parties, and utilizing more of our multiplayer techniques minimizes the need for large teams and confirms the effectiveness of a game title’s multiplayer functionality.
Intelligent minds with access to the latest technology creates great coverage in this area. We ensure that we notify the right people at the appropriate time of any areas without acceptable performance. This includes (and isn’t limited to) frame rates, memory usage, load times, and level of detail issues. Performance testing is conducted using tools as more game play is added throughout builds. Stress and soak tests are used to confirm specific inputs do not impact performance and that the game does not degrade over time. The result of performance testing indicate areas of the design that needs optimization.
This testing is done using targeted personas customary to each game. We identify likely player profiles (e.g. a casual gamer or a completionist who has to get all achievements and play all levels) and develop scenarios of how that persona might typical play the game. Then we execute those scenarios and modify testing as needed based on bug discovery and coverage telemetry metrics.
Full localization and translation testing is handled with respect to gameplay, audio, currency, globalization, and game text in addition to release procedures. Localization testers will use a subset of the original functional test script to perform a high-level test pass of the game to ensure the basic functions work from menu startup to game start and game end. All in-scope languages will implement the same high-level test script for these test passes.
Enlist people to test a game outside of the regular test team and give them focused areas to concentrate on. Coordinate bug report collection, compare to known bugs, consolidate and pass along feedback to production for action. Selective betas now test in production with limited release of new functionality to initially small targeted communities and then growing larger, using telemetry metrics to confirm viability of new release features; we assist production on planning and execution of these beta releases.