Overview

During my placement at Playstation, I got the pleasure of contributing towards the development of the testing automation for an unreleased fighting game. The studio had sent a list of automation tests and reporting that they wanted for their title, my role was to work alongside a Test Engineer to develop a testing environment which met the studio's requirements.

The studio wanted us to implement the following tests:

Test	Purpose
Character Selection	Verify every character can be selected
Move Set Validation	Verify all abilities and movements execute correctly
Costume Validation	Verify costumes do not cause crashes
Level Validation	Verify every level loads successfully
Volume Testing	Detect crashes and performance issues during long-running sessions

Existing Framework

During my employment at Playstation, the Test Automation team utilised a test automation framework written in C#. This framework handled test functionality such as hardware setup, test execution and results reporting. Our role for this automation was not developing the core automation framework itself, but to build an application which extends from Test Automation framework adding functionality and support for the unannounced fighting game.

Game-Specific Automation Layer

Due to the workloads of other engineers, I begun work on the automation solo and had complete control over the initial first stages of development for the automation.

At the start of my development for the project, I focused on implementing underlying support systems to make test development as convenient as possible. To achieve this I implemented object-oriented systems to support menu navigation (such as mode selection, character selection, and stage selection), character move execution and other utility functionality.

Additionally, we also added support for performance and memory tracking through Unreal Engine insights, which allowed us to easily record and provide test run metrics to the studio, including information such as the average frame time, memory usage or performance behaviour over extended stress tests.

Character Support

Every character was represented with their corresponding class, this class stored information such as how to select them within the character selection screen and the character's move set alongside the inputs. Due to the title's extensive roster, significant effort was required to populate and maintain character-specific move and selection data, but once populated with data, this approach presented a very easy way for tests to access any character it wanted, additionally, any maintenance, such as updating move inputs or selection logic, could be performed in a single location and automatically applied to all tests that used the character

Input Execution

Every character's move had an array on the required inputs for a move. This included information such as the delay between button presses along with allowing for multiple button inputs at the same time. Sending the actual inputs themselves to the target console was handled by DART, an application part of Playstation's SDK. DART hadn't been previously used for a scenario that requires quick and precise inputs that our test automation required, as a result we were encountering edge case bugs and were in close talks with the DART maintenance team during the development of the automation.

Move Set Mutation

Within the unreleased fighting game, almost all moves featured variations, for instance a ground version and an air version, or a square button version and a triangle button version. To avoid having to hard code ever single variation, moves allowed for "hasAir" and "hasTriangle" flags to be set. What this did is automatically generated further data for character moves, adding in air variations or triangle variations of the original command. This approach further simplified maintenance for move sets as multiple versions of the same move did not have to be written and any possible changes for the setup on air moves or alt moves could be modified from the move generation and apply to every move.

Title State and Character Move Verification

During development we encountered the issue of it being difficult to verify the title's current condition or if an action was successful. We had initially hoped that this information would be exposed to us through TTY output from the title but as this wasn't the case we had to look for alternative approaches.

We landed on using image recognition to determine what state the title is in in addition to using it for character hit confirmation. To determine states we captured a static element within each scene, we then check for if this element is present within the title's current video output. If it is then we're in the expected state for the automation, if not the title has behaved unexpectedly and we would have to handle restarting it. For hit conformation we looked at the enemy's health bar and checked whether it is full or if it's took damage by checking the colour values near the end of the bar. THis was useful for the character move set test as it let us confirm if an attack had connected.

Results

By the end of my time on the project, we had accomplished a reliable test automation which would run on command or when a build is received, execute all tests to verify functionality and stress test the title for crashes and performance indefinitely until a new build was detected. Additionally, the title-specific framework layer developed specifically for this title enables future developers to easily create and implement new tests for the automation.

Lessons Learned

While starting the project on my own was rather nerve racking at the time, it filled me with great pride looking back at the features I implemented which were deemed up to industry quality when my work was reviewed by the test engineer I was working with.

• Better time management / Underestimating development time
• Communicating to a client studio
• Further experience with the Playstation SDK
• Experience with developing on services such as Jenkins and TestRail