Metalborne: Multilevel Usability Study

Think-aloud usability test

Multivariate Testing

Expectation vs Satisfaction


I was approached by a client to try and solve a unique problem they were facing with their newly released game 'Metalborne'. The game had a high download rate, but it also had a very high churn rate. Nearly two-thirds of its user base uninstalled the game after just around 10 minutes of gameplay. The company reached out to me to determine what was the issue that was causing players to uninstall the game after such a short amount of playthrough time.  

Metalborne is an on-the-rails shooter type mobile game revolving around acquiring mecha and using them to defend a city. It involves a strategic base-building component and a third-person shooting component. As a player, you are not allowed control over your mecha's movement, the mecha will follow a predetermined path through a level while you use the touch screen to gun down enemies, like riding a roller coaster with a gun in your hand. Hence, the term on-the-rails. 

Understanding the problem

What was interesting about the problem was not the high churn rate, I mean it makes sense to conduct usability testing to determine a possible cause for this issue, but it was the fact that a large chunk of this churn was happening in the first few minutes of gameplay. According to the client, there was probably a usability issue that was causing players to get frustrated enough to want to quit early. The client had also run a remote quantitative study, where they got a group of participants to rate the game after playing through the first level and recorded an average star rating of 4 and a positive NPS score. Okay, that shows that the game should have created a positive first impression on players and does not explain the high churn. So the client believed that maybe qualitative usability testing would provide a better answer and that's where I came in.

Having conducted lots of 'first-time user experience' tests, in my experience, most users do form opinions about a game from just a few seconds of viewing the game as it starts. But, deciding to quit a game based on that opinion is something I find quite rare. It's unusual to want to quit a game this early, this usually happens after going through a bit of gameplay. Players may also quit the game early on if they realize that its genre does not suit them. But in this case, the high download rate shows that at the very least this genre was probably preferred. Since a few minutes of gameplay cannot possibly be decisive enough to cause such a high churn rate, I had a hunch that there could probably be a causal relationship between a factor in the user journey prior to actual gameplay and player frustration.

Identifying a solution

After getting hands-on and playing the game myself, I could not hypothesize a possible reason that a player would want to quit this game. So, I started to follow the journey users take before downloading a mobile game and determined that many users would have watched the promotional videos first. I accessed the play store and was viewing the promo videos when a colleague of mine happened to walk by and watch the videos. He exclaimed 'This game looks amazing, can I play it ?'. I loaded the game up and handed the mobile phone to him, he started the first mission and it took him just a few minutes before he closed the game, handed the phone back to me saying 'yeah, don't want to play it'. I was surprised and asked what made him stop playing, and he replied 'Just wasn't what I expected it to be'. It was like one of those moments where a light bulb pops up above your head, I theorized that this is most likely the main reason for causing players to want to quit. The expectation after watching the promotional videos and the satisfaction after starting the game is not matching up. 



First, we identified all the promotional videos that the users had access to prior to downloading the game. There were a total of four videos. Then we designed the study accordingly:

Construct being tested: Expectation

Main Hypothesis: The difference between a player's expectation from viewing promotional video and the gameplay experience in the first 5 minutes will cause players to want to stop playing

Sub Hypothesis:

  • Participants who played the game without watching a promotional video will not exhibit a desire to stop playing

  • The base building aspect of the game not being shown in promotional videos will deteriorate player experience

Variables: Independent Variable chosen was the promotional video with four levels. The dependent variable was the level to which the expectation was matched.


  • Between subject study

  • Both quantitative and qualitative measures

  • Random order of videos shown to minimize interference effects

  • Comparative analysis

  • Control group consisted of players who did not watch a single video

Sample size:

  • 10 participants per group

  • A random sampling of mobile game players

  • A mix of male and female participants (at least 4 female participants per group) 

  • Minimum age of 16

Test Plan:

  • Participants were divided into four groups of 10

  • Each group got to experience a single promotional video before downloading and playing the game

  • Testing conditions were set up so that participants would behave as though they picked up this game at home allowing them to stop playing at their discretion

  • Participants were interviewed and given a survey after watching the video to determine expectations

  • Participants then proceed to download and play the game

  • Participants will be given a post-interview and a short survey after quitting gameplay to determine if initial expectations were met


  • Quantitative - Survey consisting of NPS score, Likert scale, PANAS and sematic differential scales in the form of a bipolar matrix was given

  • Qualitative - Think aloud test and interview


It was observed that 40% of the participants stated that they felt like the game did not match their expectations and wanted to quit the game on the first level. Hence, the main hypothesis was supported.

All the players of the control group (not exposed to a promotional video) did not express a desire to stop playing the game. This supports the first sub-hypothesis

Many participants stated that the base building aspect of the game deteriorated from their overall gameplay experience. This supported the second sub-hypothesis.

The client acknowledged the results and completely changed their marketing campaign by redoing their promotional videos to more accurately depict gameplay. The client did experience a drop in download count after but also reported a significant drop in churn rate and an increase in player retention. Nearly every player who downloaded the game after the new marketing campaign remained and the company still reports a high number of daily active users.