Metalborne: Multilevel Usability Study
Think-aloud usability test
Expectation vs Satisfaction
I was approached by a client to try and solve a unique problem they were facing with their newly released game 'Metalborne'. The game had a high download rate, but it also had a very high churn rate. Nearly two-thirds of its user base uninstalled the game after just around 10 minutes of gameplay. The company reached out to me to determine what was the issue that was causing players to uninstall the game after such a short amount of playthrough time.
Metalborne is an on-the-rails shooter type mobile game revolving around acquiring mecha and using them to defend a city. It involves a strategic base-building component and a third-person shooting component. As a player, you are not allowed control over your mecha's movement, the mecha will follow a predetermined path through a level while you use the touch screen to gun down enemies, like riding a roller coaster with a gun in your hand. Hence, the term on-the-rails.
Understanding the problem
What was interesting about the problem was not the high churn rate, I mean it makes sense to conduct usability testing to determine a possible cause for this issue, but it was the fact that a large chunk of this churn was happening in the first few minutes of gameplay. According to the client, there was probably a usability issue that was causing players to get frustrated enough to want to quit early. The client had also run a remote quantitative study, where they got a group of participants to rate the game after playing through the first level and recorded an average star rating of 4 and a positive NPS score. Okay, that shows that the game should have created a positive first impression on players and does not explain the high churn. So the client believed that maybe qualitative usability testing would provide a better answer and that's where I came in.
Having conducted lots of 'first-time user experience' tests, in my experience, most users do form opinions about a game from just a few seconds of viewing the game as it starts. But, deciding to quit a game based on that opinion is something I find quite rare. It's unusual to want to quit a game this early, this usually happens after going through a bit of gameplay. Players may also quit the game early on if they realize that its genre does not suit them. But in this case, the high download rate shows that at the very least this genre was probably preferred. Since a few minutes of gameplay cannot possibly be decisive enough to cause such a high churn rate, I had a hunch that there could probably be a causal relationship between a factor in the user journey prior to actual gameplay and player frustration.
Identifying a solution
After getting hands-on and playing the game myself, I could not hypothesize a possible reason that a player would want to quit this game. So, I started to follow the journey users take before downloading a mobile game and determined that many users would have watched the promotional videos first. I accessed the play store and was viewing the promo videos when a colleague of mine happened to walk by and watch the videos. He exclaimed 'This game looks amazing, can I play it ?'. I loaded the game up and handed the mobile phone to him, he started the first mission and it took him just a few minutes before he closed the game, handed the phone back to me saying 'yeah, don't want to play it'. I was surprised and asked what made him stop playing, and he replied 'Just wasn't what I expected it to be'. It was like one of those moments where a light bulb pops up above your head, I theorized that this is most likely the main reason for causing players to want to quit. The expectation after watching the promotional videos and the satisfaction after starting the game is not matching up.
First, we identified all the promotional videos that the users had access to prior to downloading the game. There were a total of four videos. Then we designed the study accordingly:
Construct being tested: Expectation
Main Hypothesis: The difference between a player's expectation from viewing promotional video and the gameplay experience in the first 5 minutes will cause players to want to stop playing
Participants who played the game without watching a promotional video will not exhibit a desire to stop playing
The base building aspect of the game not being shown in promotional videos will deteriorate player experience
Variables: Independent Variable chosen was the promotional video with four levels. The dependent variable was the level to which the expectation was matched.
Between subject study
Both quantitative and qualitative measures
Random order of videos shown to minimize interference effects
Control group consisted of players who did not watch a single video
10 participants per group
A random sampling of mobile game players
A mix of male and female participants (at least 4 female participants per group)
Minimum age of 16
Participants were divided into four groups of 10
Each group got to experience a single promotional video before downloading and playing the game
Testing conditions were set up so that participants would behave as though they picked up this game at home allowing them to stop playing at their discretion
Participants were interviewed and given a survey after watching the video to determine expectations
Participants then proceed to download and play the game
Participants will be given a post-interview and a short survey after quitting gameplay to determine if initial expectations were met
Quantitative - Survey consisting of NPS score, Likert scale, PANAS and sematic differential scales in the form of a bipolar matrix was given
Qualitative - Think aloud test and interview
It was observed that 40% of the participants stated that they felt like the game did not match their expectations and wanted to quit the game on the first level. Hence, the main hypothesis was supported.
All the players of the control group (not exposed to a promotional video) did not express a desire to stop playing the game. This supports the first sub-hypothesis
Many participants stated that the base building aspect of the game deteriorated from their overall gameplay experience. This supported the second sub-hypothesis.
The client acknowledged the results and completely changed their marketing campaign by redoing their promotional videos to more accurately depict gameplay. The client did experience a drop in download count after but also reported a significant drop in churn rate and an increase in player retention. Nearly every player who downloaded the game after the new marketing campaign remained and the company still reports a high number of daily active users.