Designing and developing video games is a challenging and demanding process. Developers must create a game that is enjoyable and rewarding to a diverse audience of players to ensure the product garners financial success. Thus, understanding how different players interact and behave during gameplay is of vital importance. An accurate understanding of target players and their gameplay experience can help to identify and resolve potential problem areas during development, leading to a better player experience and arguably greater game review scores and sales.
According to the recent Entertainment Software Association (ESA) report, over the past few decades the video games market has increased as part of the entertainment industry. In addition to its rapidly growing market share, there are other factors that make video games a popular topic for exploration including new business models, widening player demographics, and new controller interfaces and platforms. These present opportunities, but also additional uncertainties. When combined with escalating design and development costs for large-scale titles, it is critical that developers ensure every game is a success in the marketplace. Put simply, the opportunity is the wider market; the challenge is that new demographics or platforms require a deeper understanding of players to appeal to this market. Generally, the industry is moving away from a single player stereotype (for instance, the stereotypical image of a teenage boy who plays games for many hours per day on a game console) to the new reality in which multiple ubiquitous gamers play across locations and mobile platforms. Modern video games offer different ways of interaction and the potential to deliver a better player experience.
The emerging field of Games User Research (GUR) seeks to improve the gameplay experience by conducting usability and UX evaluation on under- development titles. GUR is a field that borrows user research methods—such as behavioral observation, interview, questionnaire, and heuristic evaluation—from Human-Computer Interaction (HCI) and psychology. Although user research methods have made progress in understanding the usability of productivity applications, applying these methods to video games is still a challenge for practitioners and researchers. Due to the specific characteristics of video games (Table 1), most of the well-established user research methods cannot be used in the same way for formative evaluation of the player experience.
Table 1. Differences between games and productivity applications (based on Pagulayan et al in “User-centered Design in Games” (Handbook for Human-Computer Interaction in Interactive Systems, 2003).
| Games vs. Productivity Applications | Examples |
| Process vs. results | The purpose of gaming is usually in the process of playing, not in the final result. |
| Defining goals vs. importing goals | Games (or gamers) usually define their own goals, or how to reach a game’s goal. However, in productivity applications, the goals are usually defined by external factors. |
| Few alternatives vs. many alternatives | Games are encouraged to support alternative choices to reach the overall goal, whereas choices are usually limited in productivity applications. |
| Being consistent vs. generating variety | Games are designed to provide a variety of experiences. However productivity applications are meant to be consistent in the user experience. |
| Imposing constraints vs. removing or structuring constraints | Game designers intentionally embed constraints into the game loop, but productivity applications aim to minimize constraints. |
| Function vs. mood | Productivity applications are built around functionality, but games set out to create mood (for example, using sound or music to set a tone). |
| View of outcome vs. view of world | Gamers usually play a role in a game world such as race car driver, soldier, warrior, etc. Productivity applications rarely have a point of view. |
| Organization as buyer vs. individual as buyer | Individuals usually buy games, but productivity applications are often bought by organizations. |
| Form follows function vs. function follows form | Gamers tend to welcome innovation while users of productivity applications tend to be cautious about adopting innovation. |
| Standard input devices vs. novel input devices | Games usually explore possibilities to use novel input methods, such as motion capture or biofeedback, in addition to standard input devices. Productivity applications mostly rely on a mouse and keyboard. |
Hence, user research methods have been adapted and evolved to better fit with game development needs. These methods aim to provide a mixture of qualitative and quantitative approaches for games user researchers to choose from, depending on their user study setup and needs. Identifying the best mixture of these methods, and blending the results from each into a meaningful report for game developers, is one of the current challenges facing UX and usability evaluation in games. At the GUR Summit 2012, Bill Fulton discussed an evaluation framework for conducting game usability and UX studies with a focus on formative evaluation (Table 2).
Table 2. Criteria for applied usability and UX evaluation on games (based on http://vimeo.com/groups/gursig/videos/26733185)
| Criteria | Description |
| Representative | Selected methods and recruited participants must correctly reflect user testing needs and outcomes. |
| Accurate | Results should reflect user testing assumptions and include multiple sources of supporting data. |
| Specific | Methods selected for conducting the test need to deliver precise and specific results. For example, they cannot state that a game is not good without indicating why or identifying the problems. |
| Timely | User test findings should be delivered in a timeframe that matches the game development cycle. |
| Cost-effective | There must be a return on investment or value added to a game that justifies the cost of conducting user tests. |
| Actionable | Results need to be delivered in an actionable and applicable format. The quality of results is directly affected by the chosen methods and analysis approaches. |
| Motivational | Presented results should motivate game developers to take action on them. Game developers should believe in and fully understand the results. |
As Table 2 shows, evaluation methodologies influence the success of every GUR study in a significant way. However, the usual methods of determining the player experience are based on self-reporting methods. Although these are relatively easy to conduct and can potentially provide a rich source of data, they rely heavily on players’ awareness of the experience, ability to recall it, and their cognitive skills to explain that experience.
In order to address these issues, Games UX researchers are using a number of innovative techniques to enhance user studies, better understand players’ experience, and develop new methods of communicating user test findings to the game development team. There is an increased interest in approaches such as physiological measurements and behavioral analytics to supplement traditional UX evaluation methods for games.
Physiological measurements or biometrics refer to the capture and analysis of signals directly from the user’s body, often using skin-contact sensors that show how different users react to events or stimuli on the screen. Since physiological evaluation measures visceral biological responses, they are instinctive and hard to fake. Game user researchers also leverage the use of behavioral analytics to better understand players’ actions and behavior by collecting large amounts of data regarding player movement, interactions, or position (progress) in the game world. These data can enable the analysis of long-term player behavior, which would be particularly useful for ensuring balancing issues. These issues may not be easily detected by classic user testing approaches, especially during short test sessions.
Another challenge in studying the player experience is tying together the results of quantitative and qualitative user research data. For example, combining data from players’ physiological measures with questionnaire or interview results and in-game movement data into a single report is not straightforward because the underlying data is often in different formats. Visualization techniques can facilitate the understanding of relationships among these data sets, and various visualization techniques have been introduced in game UX. However, most of these techniques focus on displaying large amounts of data captured directly via telemetry without integrating qualitative or contextual data about players’ emotional experience. Hence, a potential advancement would be to develop a new visualization technique that can triangulate the above mentioned mixed data sources.
Figure 1 shows one approach that took a step towards solving this challenge. In the visualization, an excerpt from a level of a popular 2D platform game is displaying players’ movement data, their verbal comments from interviews, and their physiological data in one combined visualization. The overall hue of the paths gives an impression of the arousal of the players (measured utilizing a physiological sensor) at certain points of a level. For example, in area A the paths are mostly yellowish, which indicates low arousal. This is a safe area as there are no enemies or gaps that can lead to immediate death in case a jump goes wrong. In contrast, the paths in area B are mostly red, which signifies high arousal, presumably because players had to cross two gaps while simultaneously fighting an enemy located on the platform in-between. Similarly, a huge gap in area C that needs to be crossed with a long jump, also caused a higher level of arousal in players. Interestingly, judging from the player trails, the jump itself seems to be quite easy as none of the players fell to their death at that point. In this sense, the visualization provides insights to designers regarding how they might tweak the pacing of the game. For example, it might suggest that they alternate high intensity jump areas with periods of recovery.
To summarize, game development is a complex, costly, and lengthy process that requires developers make creative and novel design decisions while satisfying new, diverse markets. It’s critical for developers to receive feedback from their target audience on their experience with games and how they might be improved. However, player behavior is often studied based on subjective reporting and observational approaches, and may not include the emotional component of gameplay. Approaches such as physiological measurements and behavioral analytics have the potential to provide continuous monitoring of a player’s affective state without interrupting the player. Combining these measures with other user research data sources, and identifying the elements to change, is a lingering challenge that must be addressed to enhance design decisions in game development
[bluebox]
More Reading
Beyond Thunderdome: Debating the effectiveness of different user-research techniques. (Vimeo video, without captions) Presentation by Mike Ambiner and John Hopson.
Mirza-Babaei, P., Wallner, G., McAllister, G., & Nacke, L. (2014) Unified visualization of quantitative and qualitative playtesting data. In CHI EA ’14. ACM, 1363-1368.
[/bluebox]
