Computer Games and Emotions

Petri Lankoski

Authoritative version is published in Sageng, Fossheim & Larsen (eds.), The Philosophy of Computer Games, Springer, pp. 39–55,  DOI=10.1007/978-94-007-4249-9_4.

Introduction

An intriguing question in the philosophy of fiction is on how can we be moved by the fates of the fictional characters or how we fear a fictional monster? This question, in the context of literature and film, has been addressed, for example, by Lamarque (2004/1981), Carroll (1990, pp. 61–96), and Walton (1990, pp. 240–289). This same question is relevant in the context of computer games: how can players be afraid in the game events when obviously, for example, a monster in a horror game cannot threaten the players?

Hjort and Laver propose that the connection between emotions and art is even more fundamental:

It is generally assumed that art and emotion are inextricably linked, as is shown by even the most cursory account of the history of critical thinking about music, painting, literature, or theatre (Hjort and Laver 1997).

Carroll (2006) agrees that many aesthetic experiences provoke emotions, but makes a reservation that emotions are not necessary for all aesthetic experiences (p. 93). However, if emotions are typical in aesthetics experiences, understanding how emotions and gameplay relate is relevant in understanding the aesthetic experience of playing computer games.

According to Damasio (2005), the important function of emotions, in general, is to guide decision-making and attention (pp. 173–177). This implies that emotions are a vital part of computer games, as games are typically about decision-making (I return to this below). In addition to emotions, the decision-making is guided also by the goals, because they give means to reason and determine which outcome is more advantageous in a given situation. Without a preferred outcome, the decisions are meaningless.

While it is not my intention to provide the definition for computer games, some consideration of this is needed to argue why goals are important when discussing computer games. Goals are very typical components of games. Various authors have highlighted the importance of goals in games by making goal-directed conflicts as an important (or even defining) structural component (Costikyan 2002, pp. 11–17; Salen and Zimmerman 2004, p. 96; Järvinen 2008, pp. 53–54, 69–72; Bj€ork and Holopainen 2005, pp. 14–19). For example, SimCity (Maxis Software Inc. 1989) is considered as an example of a borderline case of games because it does not have goals (Juul 2005, pp. 28, 47, 199). However, the game has two scoring mechanisms (city score and public opinion) and it has a (non-fatal) failure condition (the players are unable to act when they have no money). Hence, the system (e.g., the scoring) implies goals and enables players to judge what is more advantageous.

Emotions are also important in the social domain—when we are interacting with others. Empathy with characters is also important in computer games: Morrison and Ziemke (2005) argue that players have empathic reactions with computer game characters (see also, Perron 2009, pp. 125, 140–141). They ground their argument to neurological studies. I elsewhere argue that the players’ engagement with character-based games can be understood via modes of goal-related engagement and empathic engagement and engagement with NPCs through empathic engagement (Lankoski 2011; Lankoski 2010, pp. 21–28).

However, the above-mentioned perspectives are not enough: It has been demonstrated that negative events in a game can induct positive emotions to players, which might be explained by the cuteness or beauty of the presentations of the negative consequences (Ravaja et al. 2006). Moffat and Kiegler (2006) show empirical evidence to show that music influences a person watching a cut-scene. It has been argued that that the music influences the playing experience as well (Ekman 2008, pp. 22–25). One needs only to turn off the sounds from the Silent Hill 3 (Team Silent 2003) or Thief Deadly Shadows (Ion Storm 2005) to see how the games lose their emotional impact. Music and beauty are similar to empathic engagement, because they cannot be (fully) explained by the goal-related emotion theories.

The approach in this essay is based on models and theories in the cognitive science. I propose that to understand why games are engaging, we must understand how goals, events, sounds, and graphics in games and affects are connected and how the different emotion mechanisms interoperate. To achieve this, I use various cognitive theories and concepts (1) as analysis foci to categorize game events and choices available for a player and (2) explain the probable impact of the game events, objects, and environment.

In this essay, I argue that various perspectives are needed in order to understand the players’ playing experience(s). My main goal in this essay is to study how players engage in games emotionally, namely how the emotions of the players and the gameplay is connected. To do this I look at how different aspects of the game relate to the emotional experience:

1. How the structural features (e.g., goals, action possibilities, events) of the game relate to the experienced emotions.
2. How visuals and sounds modulate the emotional experience.

For this purpose, I describe game structures, visuals, and sounds and I use the cognitive emotion theories to predict likely emotional associations for these. I build on Antonio Damasio’s (2005) and Power and Dalghleish’s (1997) theories on emotions. Power and Dalgleish’s theory is based on the emotion theory by Johnson-Laird and Oatley. These theories posit the same linking between goals and goals status evaluation, but Power and Dalghleish’s (1997) theory provides more developed account of disgust. Damasio acknowledges Johnson-Laird and Oatley’s theory as a compatible one with his theory (Damasio 2005, p. 201). The basic emotion theory does not offer an answer to the question: why people find beautiful things pleasurable or why music elicits different emotional reactions.

To answer this I propose that the theory of processing fluency is useful (Reber et al. 2004) in terms of understanding the pleasure relating to beauty. This theory of processing fluency has similarities with Sloboda and Juslin’s (2001) theory on how music elicits emotions. Sloboda and Juslin’s approach is used as a starting point in this essay to study emotions and sounds. I need to stress that these theories are not part of the same theory as Power and Dalghleish (1997), but, the theories of Reber et al. (2004) and Sloboda and Juslin (2001) are all cognitive appraisal theories and, hence, have the common premises. Hence, I believe that the theories can be used to create a holistic theory of gameplay emotions.

The structure of this essay is as follows: First I look at the basic mode of the engagement: goals and emotion. After that I focus on how the visual and auditive aspects (that are not explainable with the goal-related model) of games contribute to the emotional experience; the aspects that are discussed are facial and bodily expressions of a character, visual beauty, and sound and how these aspects relate to emotions.

Goals and Emotions

First, I discuss the nature of goals in games. After that, I describe games using the above-presented basic emotions theories to link probable emotions within the event structure of a game. My goal here is to isolate and exemplify the features that are relevant in goal-related engagement.

Goals

Games propose goals to a player. In some games, such as Tetris (Pazhitnov 1985), the player needs to accept a proposed goal in order to keep playing; ignoring the goal will lead to a prompt game over. On the other hand, a game can have a goal structure that regulates progress in the game: in order to get to the next game level a player needs to attain a goal.

Grand Theft Auto: Vice City (Rockstar North 2002) belongs to the group of games in which players can choose to ignore the regulating goal structure without much penalty and generate their own goals. The players have a rich game world to explore with potential challenges, even without following the goal structure. Below, to simplify the argument, I consider only situations in which players accept the goals made explicit by a game: in order to keep playing do A or in order to progress do B. In any case there are also the player’s goals for playing a game, which can influence the generation of sub-goals.

The sub-goals are goals inferred from the more generic goals (of a game, game level and playing) in relation to a specific situation. For example, in Tetris, a player needs to generate a sub-goal for placing a tile. Sometimes the sub-goals generated are impossible: either it is not possible to complete the sub-goal, or the sub-goal does not lead to progression toward the main goal. Then the player needs to change his hypothesis of how to reach the main goal and generate new sub-goals.¹

In some games, there are multiple ways to complete a game level and each solution requires different sub-goal generation. One game that shows this is Deus Ex (Ion Storm 2002). In the game, usually a player can choose whether to shoot her way through a level or sneak past enemies. Therefore, a player’s goals, skills, and preferences can have a great impact on sub-goal generation (the choices will require different breakdowns of the sub-goals). Also, failure in sneaking past the enemy will change the situation and trigger a reevaluation of the situation, which possibly requires a new set of sub-goals (or replaying from a game save).

Basic Emotions

Antonio Damasio (2005) distinguishes primary and secondary emotions. Primary emotions are pre-organized speedy evaluation of a situation where the body state and cognitive processing is altered in the manner that fits to a situation, for example, fear triggered by an entity X includes physical reactions and preparation to action (or in some cases freezing in fear). One does not need to recognize the entity X as, for example, a bear or snake in order to fear it. All that is required is that early sensory cortices detect and categorize the key features of the entity as something dangerous. However, with speed comes a possible inaccuracy such as categorizing a picture of a snake as a real snake. The responses can be innate or based on one’s history of interaction with the environment. Notably, later the inferences will shape people’s responses of a situation, but emotions including body responses and changes in cognitive processing have already triggered before that (Damasio 2005, pp. 125–164).

Secondary emotions are triggered by conscious and intentional consideration of a (hypothetical) situation. The as-if consideration triggers the automatic and involuntary changes in the body state and in cognitive processing. For example, fear is triggered when considering a risky option or encountering a wolf (Damasio 2005, pp. 134–139). Damasio’s theory proposes that people categorize events as positive (correlating positive emotions) and negative (correlating negative emotions).

Mick Power and Tim Dalgleish (1997) suggest an emotion model where different basic emotions are linked to different goal status evaluations. These are shown in Table 4.1. Interestingly, the basic emotions and mechanisms that trigger them seem to be intercultural (Ekman 1999a). Thus, in some degree, discussion in this paper should be applicable across cultural borders.

Table 1. Basic emotions, goal status evaluations that the basic emotion is linked to, and typical physiology/action tendency of the emotion.
Source: Power and Dalgleish (1997, pp. 413–425), Oatley (1992, pp. 55, 350–363)
Note: According to Paul Ekman, all basic emotions are associated with typical facial muscle
activations and typical expressions (e.g., Ekman 1999b), but these are not included in this table

Complex emotions are derived from the basic emotions and include with the appraisal of context or combination of different basic emotions. For example, worry is about an unwanted outcome that might happen in the future (Power and Dalgleish 1997, pp. 413–421). In these terms, the horror genre draws more from disgust than from fear.

Power and Dalgleish (1997) talk about two routes to emotion: the appraisal of a goal related event (an event can be external, e.g., a snake; internal, e.g., stomach pain; or propositional, i.e., thoughts and reasoning) and a direct access (an event triggers an emotion because of habituation) (pp. 415–427). Damasio’s primary emotions could be said to include these both. In addition, Power and Dalghleish (1997) propose that the emotion process has a propositional level in which reasoning induces emotions via either appraisal or direct access (pp. 415–421). This propositional-level process matches with Damasio’s secondary emotions.

These theories propose that the emotions that a player experiences while playing are real (in contrast to be make-believe); for example, we can be afraid of something that exists only in a game. This is in contrast to a well-known theory by Kendal Walton (1990) that proposes that a player (of make-believe) would only experience quasi-fear: he argues that we cannot really be afraid of something if we know it is fictional (the monster cannot physically hurt us). Here, I disagree also with Grant Tavinor who builds on Walton’s theory and argues:

[T]he Big Daddies in BioShock are so threatening that the players must steel themselves before an encounter [. . .] This is because, fictionally, the player-character and the BigDaddy do “exist” in the same ontological game world (Tavinor 2009, p. 142).

Instead, I propose that BigDaddy can be frightening, because it threatens the player’s goals when it (fictionally) threatens the player character. Abstract games such as Tetris operate predominantly with basic emotions.²

Next, I trace the emotional implications of different events and goal-status evaluations in Tetris using the above-presented emotion theory to demonstrate how emotions and gameplay are connected (refer to Table 1 for goal status evaluations and their relation to emotions). The structure of Tetris implies following an emotion sequences (an example is provided in Fig. 4.1). In the sequence, fear is experienced when a dropping tile threatens a goal and this fear implies increased arousal and vigilance that helps to maintain the control in the game. The fear is followed by happiness if the tile is places as planned or sadness if the action fails.

The complex emotion worry is a combination of fear and anticipation of an uncertain threat (Power and Dalgleish 1997, p. 207), and can be experienced when reaching a sub-goal that requires certain kind of tile before it is too late. This worry can work as a warning sign that encourages the creation of a new and less risky sub-goal. This series is repeated until a final failure, which is subsequently evaluated either as success resulting in happiness or as failure resulting in sadness depending on whether a player’s goal of playing was fulfilled or not, or alternatively as anger if the game is evaluated to be too hard.

In Tetris, the high score list has an important role in the final evaluation: it offers concrete grounds for the appraisal of one’s performance in relation to the history of the playing performances. Naturally, the description above is not enough to catch the playing experience, but the description sketches the basis (that should be rather player independent) of the experience. The extended appraisal of the situation with current emotions can lead to complex emotions: for example, a player can feel pride when succeeding really well when evaluated in relation to the performances in the past.

Fig 1. An example of a situation in Tetris (Pazhitnov 1985). The player’s sub-goal is illustrated
on the left hand side (1). When the goal is reached (2A), the player experiences happiness.
If the goal fails (2B) the player experiences sadness; but if the failure is as the one presented in
2B, the player is likely to experience anger (or regret, which is an anger-based complex emotion)
as the failure blocks the feasible goal(s) and makes the game harder

Conversely, disgust has a strong link to representations. Prompting the feeling of disgust is used in horror games. Carroll (1990) argues that the monsters and the peoples’ reactions—such as nausea and revulsion—to the monster have a key role in the horror genre (pp. 13–42). Ekman and Lankoski (2009) propose that disgust and its relation to the contamination can be used to explain the horror experience in the computer games and especially in Silent Hill 3 and Fatal Frame (pp. 181–182, 190–192).

Disgust requires that the players consider something as contaminating. In this, the horror games often trust the above-described mechanism that causes primary emotions, for instance, the idea of contamination via rotten food relates to disgust, whether the food is actual or fictional (see, Power and Dalgleish 1997, p. 347).³ In addition to the threatening goals, the monster’s role is to generate disgust.

In Silent Hill 3 monsters are filthy and decayed or are partly insects; insects and blood are typically associated with disgust. In addition to a digestion-related effect, the game presents a more abstract form of contamination. A non-player character says in the last part of the game that the monsters are human beings and Heather, the player character, is only sees them as monsters. With this, the game tries to induce guilt—guild is a disgust-based emotion (see, Power and Dalgleish 1997, pp. 355–358)—by psychologically contaminating⁴ the action of killing monsters by proposing that the player is controlling an insane killer (c.f., Lankoski 2011).

When the player character goal’s for the game are derived from the fictional goals of the player character, the player’s emotions and player character’s fictional emotions are correlated; the emotions are a result from the goal status evaluation, as discussed above, but the emotions while playing relates to the fictional emotions of the character because of goals are correlated. I consider this similar to empathy as discussed below, but as the goal status evaluations of the player’s own goals are the source of emotions, it is vital to discuss this separately (c.f., Lankoski 2011).

Thief 2: The Metal Age (Looking Glass Studios 2000) belongs to games in which the goals are in the primary form to create an emotional connection between a player’s emotions and the player character’s fictional emotions. For example, a goal in the Shipping. . . and Receiving mission is to steal at least loot forth 850 gold. Guards during the mission put the goal of a player (and the fictional goal of the player character) in danger and thus their presence implies fear; finding loot relates to happiness, failure to steal the loot implies sadness; and guards blocking (making impossible) a goal suggest anger. There are also obvious disparities between the players’ emotions and the fictional emotions of the character in the given situation: for example, the guards are a physical threat to the character but not to the player; outstandingly, the player’s goal (to keep the character alive) and the fictional physical threat would both relate to fear.

I hope that I have demonstrated how the goal status evaluations of different game events influence the player’s emotions. As seen, basic emotions can relate to rather abstract events and the presentation does not have significant impact on the appraisal. However, disgust is tightly connected with the presentation, and in some cases the same game object (enemy) would relate simply to fear, but the presentation can change the core emotion to disgust. Nonetheless, the goal status evaluations cannot explain—even the associative link of the direct access—why we find watching beautiful things pleasurable or how sounds and music elicit emotions. Next, I propose some answers to these questions.

Presentations and Emotions

First, I look at how the representation of the emotional expressions of game characters is linked to the emotions experience of emotions by the player. After that, I trace how beauty relates to happiness. Last, I study how sounds modulate and regulate the emotional experience.

Empathy

Following Jean Decety and Philip Jackson (2004), Empathy is used to refer to mechanisms that correlate ones own affects and the emotions expressed by others (p. 71). Empathy can happen regardless of whether another is an actual person or character from a film or game, including characters such as Mickey Mouse or human-like animals in Sly 3: Honor Among Thieves (Sucker Punch Productions LLC 2005; Morrison and Ziemke 2005; Lankoski 2011; Pan and Slater 2007).

I have elsewhere argued that understanding game characters depends on the same processes as understanding other people: mimicry and empathy. Mimicry refers to involuntary automatic reactions to motor actions and emotional expression, such as mirroring the expression. This mirroring can range from small muscle activations to mirroring posture and facial expression. With the mirroring expression, emotions are also mirrored to some extent (Lankoski 2011). There are evidences that suggest this mapping happens at least with touch and pain and with the basic emotions of disgust, fear, happiness, and anger (Morrison and Ziemke 2005; Dimberg et al. 2000). Empathy includes also character simulation⁵.

Character simulation is as-if reasoning where one tries to figure out what the other would do in a situation. Simulations, like other forms of as-if reasoning, can also trigger emotions via the appraisal of the goal status or direct access routes; Damasio links the as-if reasoning to secondary emotions (see above).

Some games, for example, Beyond Good and Evil (Ubisoft Montpellier Studios 2003), seek to trigger empathic reactions with non-player characters and player characters (see Fig. 2)⁶. Here the child’s expressed fear is to motivate the player about the importance of saving the child. The player character’s expression of anger is building motivation: the character is reacting to the situation and is getting ready to act. Mirroring this expressed anger implies that the player’s action readiness is likely to rise.

Fig. 2 A captured child screaming for help and reaction of Jade, the player character in a cut-scene in Beyond Good and Evil (Ubisoft Montpellier Studios 2003)

Taking control from the player may block the player’s sub-goals thus causing frustration (see above). This happens in Silent Hill 2 (Team Silent 2002) when the game takes control from a player and starts a cut-scene, in which the player character, James Sunderland, is locked in a room with monsters by a nonplayer character. This leads to a playable scene, in which the player is forced to defeat the monsters. When the first cut-scene appears for the first time, the event and following fight is a surprise and challenge, but if the players do not defeat the monster they are forced to trigger the cut-scene again. This forced replay can lead to frustration (especially if the players do not trust the non-player character in the first place). Here, I suspect that anger or frustration is not intended, and works against the playing experience. If the player grows too angry due to this goal blocking, they are likely to quit playing.

Fig. 3 In Diner Dash (gameLab 2005) the player need to monitor expressions and hearts (satisfaction meter) of the customers in order to prioritize the service

Empathy and judging character, however, requires that the player have a focus on the character. People can regulate their empathic reactions toward others by concentrating their focus on something else. Empathy can also be suppressed if the players are occupied with a cognitively demanding task (Eimer et al. 2003). However, if a given task requires focusing on the emotional expressions of the others, empathy is likely to occur, as it is needed in making a judgment of another person or character. This is exploited in Diner Dash (gameLab 2005), in which a player controls the waitress Flo. In the game, the player seat customers and guide Flo to taker orders, bill, and clean tables. If the customers, for example, need to wait too long they get angry and leave without paying their bill. Therefore, the player needs to monitor the expressions constantly in order to prioritize tasks and maximize income.^{7 The expressed anger is likely to make the player feel bad about letting the characters wait while the aggressed happiness heightens the happiness of success. Emotions via empathy might clash with the player’s desire to optimize the goal and scores, but this optimizing might create more tension in them because of the clash between empathic and goal-related emotions (Fig. 3).}

Beauty

Fear and disgust are associated with images representing something visually disgusting or fearsome, and this can be explained by the basic emotion theory, but that theory cannot, as pointed out above, explain why we experience happiness when perceiving beauty. Empirical evidence implies that people react with happiness to beautiful images (Reber et al. 2004; Paradiso et al. 1999).

Reber, Schwarz, and Winkielman argue that the perceived beauty of the artifacts relates to the features of the artifact (in relation to psychological mechanisms) and the perceivers’ past experiences:

1. Reber et al. points out that there are evidences that suggest that the judgment of beauty relates to the amount of information: simple (or even caricatures in some cases) are considered more beautiful than complex ones; more ambiguity, complexity, in cubist paintings are considered to influence negatively to the evaluation of beauty. Relating to this, symmetrical images or artifacts are found to be beautiful. Lastly, visual clarity contributes to the judgment of beauty.
2. Reber et al. maintain that repeated exposures to stimuli and learning of stimulus structures are know to result in evaluations that are more favorable. Also, prototypical forms are preferred over those that are non-prototypical (Reber et al. 2004).

Notably, according to Reber et al., if stimulus is irrelevant in relation to a current task and the task requires most of the attention, the perception is very likely ignored and consequently is not evaluated as beautiful (Reber et al. 2004). In order to enjoy the beauty of environment, games such as Ico (Sony Computer Entertainment 2002) or Shadow of the Colossus (Team ICO 2006), a player is given time to admire the beauty of landscapes before or in-between actions that demand focusing on the decision-making or motor actions. The beauty of the environment in these games is likely to be a source of happiness, and players are likely to continue to explore the environment when they experience happiness also in the parts of the game that do not contribute to goal-related engagement. Typically, the beauty of the environment is not the only reason to keep playing, but there is also curiosity: what the players have not seen. However, the beauty and happiness relating to the beauty can be a motivating factor behind the curiosity.

Some aspects of beauty judgment of a person seem to be relating to the above-mentioned framework: physically symmetrical people are considered more beautiful than those who are non-symmetrical, and stereotypes of beautiful features explains some variation in features that are considered constituting what is beautiful.

Notably, according to Braun and Bryan, female waist-to-hip and male waist-toshoulder ratios relate to the beauty judgment by the respective opposite sexes. Males prefer hourglass shaped females and females prefer V body shaped males (Braun and Bryan 2006). These beauty indicators are well exploited in games—as, for example, Jade in Beyond Good & Evil (Ubisoft Montpellier Studios 2003), Lara Croft in Tomb Raider: The Angel of Darkness (Ubisoft Entertainment 2003) and the prince in the Prince of Persia: Sands of Time (Ubisoft Montreal Studios 2003) show; especially symmetry is used.⁸ In addition, baby-like features seem to be considered as beautiful (Isbister 2006, pp. 10–12, 232). This is utilized in, for example, Super Monkey Ball Deluxe (SEGA Corporation 2005), where the features of the exaggerated faces remind the key features of baby faces. Beautiful characters are pleasurable to watch, and hence, characters can keeps the players engaged with the game, and they are likely to care about the fate of the character (continuing their goal, because of happiness associated with the characters).

Sounds

Next, I consider how the sounds will influence the emotions of playing. I consider sounds in the perspective of emotion theory of music as the view will enlighten some effects of sounds that are not possible to handle via the above-mentioned goal related model. Just as with empathy and beauty, sounds and music works in conjunction with the other aspects of the game and modulate the emotional experience.

It has been demonstrated that music and sounds can influence the inferences on events, (Moffat and Kiegler 2006; Zehner and Lipscomb 2006; Molholm et al. 2002) but if music does not break expectations the music is not likely to be remembered separately (Ekman and Lankoski 2009, pp. 187–188). Moffat and Kiegler have presented a study where a game trailer of Alan Wake by Remededy is combined with different kinds of music (sad, angry, happy, and fear) and some impressions where collected by asking questions on whether informants agreed on statements.

The informants who heard angry music agreed to the statement Alan Wake has a gun in his pocket significantly more than the informants who heard other type of music (Moffat and Kiegler 2006). Similar influence on the player’s judgment on their affects has been reported with a play-sequence elsewhere (Zehner and Lipscomb 2006).

Sloboda and Juslin (2001) build a theory of emotions in relation to music from cognitive emotion theories that assumes that emotions involve the appraisal of an event. They argue that music can be appraised as an event and be a source of emotions in that way (pp. 90–91).

They argue Sloboda and Juslin argue that in music there are structural characteristics that relate to the emotions. They suggest:

These characteristics include syncopations, enharmonic changes, melodic appoggiaturas, and other music-theoretic constructs, which have in common their intimate relationship to the creation, maintenance, confirmation, or disruption of musical expectations (Sloboda

and Juslin 2001, p. 91).

Sloboda and Juslin (2001) propose that part of the emotions experienced relates to these features, that they call intrinsic (these features refer only to other musical events and structures). They also distinguish extrinsic emotions that can be further divided as iconic and associative. The iconic relationship refers to that musical structure that has some kind of resemblance with an event or agent triggering emotions: for example, fast and loud music relates to emotions with high arousal. Iconic features are non-arbitrary, and they relate to specific musical structures. Finally, they recognize associative emotions⁹ that are arbitrary and relate to the experiences of the person. Human memory seems to associate certain kinds of stimuli (for example, sound and smell) of an event with emotions that experienced during the event. It seems that music can trigger emotions that way (pp. 91–96). In many games, the soundscape consists of traditional music, but for example in Silent Hill 3, the soundscape is an amalgam of melodies, ambient sounds, static noise, silence, pulsing squeaks, and even the dramatized sounds of footsteps. By combining different kinds of sounds with traditional melodies Silent Hill 3 breaks musical expectations, which is a source of negative emotions such as fear and uneasiness the player may experience. The raised action readiness relating to the music maintains tension even when there is no imminent threat. This prolonged tension is released with happiness when the player successfully confronts the game monsters; this contrast to tension and happiness is likely to heighten the sense of happiness of the success.

In a game, the associative emotions that might be arbitrary in the beginning are not likely to stay as such: musical themes are linked to certain kinds of events–dangerous situations, battles and triumphs that have their own special theme. Thus, it is likely that the theme gets associated to a certain event and the emotions are more consistent than in general through the repetition. The danger melody uses intrinsic or iconic features that relate to fear. In addition, the connection between the theme and event is learned. For example, in Dark Chronicle (Level-5 Inc 2002) and Silent Hill 3 the presence of a hostile non-player character is communicated with music, and the player can often hear the music before seeing the hostile character. Moreover, if the music does not relate to fear at first then it can be argued to be associated to that during the gameplay.¹⁰ Forewarning and fear incline toward raised action readiness and enable the player to plan the encounter.

However, a theme might not get associated to an event only one way (e.g., as fear), but instead turn out to be the source of annoyance if the theme does not endure multiple repetitions, as is the case with the danger music of Dark Chronicle.  Annoyance is an anger-based emotion (Power and Dalgleish 1997, p. 322) and in this case, arousal that should relate to fear is cognitively labeled as something else, namely annoyance. This annoyance might make the continuation of play less likely.

The sounds can be associated to a character, an event or place and conjure up impressions and affects (as described above). Buffy the Vampire Slayer: Chaos Bleeds (Eurocom 2003) is a game based on the television series Buffy Vampire Slayer (1997–2003). The game uses the television series theme to evoke the impressions and affects experienced when the player was following the television series.¹¹ The voice actors of the game characters are the same people who acted in the series, with the exception of the characters Buffy and Willow. The voices associate the game characters with the series strongly. This association is also likely to trigger emotions that the player has experienced when watching the series. When the player is a fan, these associations are like to increase an attachment to the game.

If cognitive load is low, the player has time to pay attention to the characters and environment and enjoy the beauty of it. However, it seems that music and sounds can shape the emotional experience despite high loads on visual perception, action and decision-making. However, unexpected sounds (or other signals) can direct attention as the human perception system is tuned to notice them (e.g., Reber et al. 2004). Despite that, the interpretation and emotion processing from music might still happen in high-spaced action situations; it seems that visual and sound processing happens, at least, partly in parallel fashion (see Molholm et al. 2002). Thus sounds in a game can have an important role in guiding emotions and attention.

Conclusions

Why to study emotions? As I indicate in the Introduction, the cognitive theory of emotions and their relation to gameplay might be a key to understand aesthetic experience of games and games as fiction.

I have argued that we need multiple perspectives to understand gameplay emotion and the playing experience. However, I have not presented a complete theory of how emotions and gameplay are connected. Including more games in the study is likely to add features that should be taken into account in terms of the playing experience. An issue that I have not considered that is that why fear and disgust can engage us. Those emotions are typically associated with avoidance behavior, but my above analysis indicates that fear is present in many different games and disgust in horror games. Yet, it seems that these emotions seem to be crucial for an aesthetic experience.

In conclusion, I have described how, by analyzing the components of a game, (including all kinds of events containing emotional expressions), it is possible to perceive how the player’s emotions and the gameplay are connected. Markedly, these connections are likely connections, but an emotion does not necessarily happen when the features are present, because, for example, attention and personal history influence how the situation is experienced. I have highlighted the role of goals and other structural features of the game system, characters, beauty and music in the emotional experience of gameplay, and show how they contribute to the experience. My main novel contribution in this essay is the detailed proposal on how these elements shape the player’s emotional experience. Moreover, I have argued that emotions are integral to the playing experience.

Endnotes

1. I am basing this discussion on ideas presented by Walton (1990, pp. 138–187) and Shaun Nichols (2004). However, my argument here does not rely on their theoretical premises.
2. Tetris is a game in which differently shaped blocks fall until they hit the bottom. The player can rotate a falling block and move it sideways until it hits the bottom. If the blocks are placed on the bottom so, that there are no holes, the row(s) vanishes. The game ends if the pile of blocks reaches the top of the play area.
3. Notably, Power and Dalgleish (1997) remarks that disgust is often confused with fear when a situation involved insects or reptiles (pp. 346–347).
4. Psychological contamination can be something that is not digestive-based. Power and Dalghleish (1997, p. 345) refer to a questionnaire study in which people judged the idea of wearing Hitler’s sweater being a one of the most disgusting options.
5. It has been argued that the simulation is used when we try to predict and understand the other people in our daily encounters. The simulation is agued to have a role on understanding the behavior and emotions of o film, or literary characters (Currie 1995, pp. 235–237; Smith 1995, pp. 17–35).
6. The example in the figure is from a cut-scene, but the characters also show emotional expressions in the playable scenes.
7. The players can also monitor the hearts, but I suspect that the facial expressions are easier.
8. Using symmetry might be due to economic factors, as the symmetrical characters are easier and faster to model: one can create only the left or right half of the model and let the software create the other half (by mirroring the created half).
9. The associative emotions here are analogous to the association of emotion and event in direct access route, see Sect. Emotions above.
10. Here, for the sake of simplicity, I use music to refer also to the static noise used in Silent Hill 3, as it is a composed piece for that function. I wish not to go deeper into the question whether the use of static noise in the game constitutes music or not (but if John Gage’s 403300 is accepted to be music, my shortcut is not a shortcut).
11. Naturally, this applies only to those players who have been following the television series.

Bibliography

• Björk, Staffan, and Jussi Holopainen. 2005. Patterns in game design. Hingham: Charles River Media.
• Braun, Margaret F., and Angela Bryan. 2006. Female waist-to-hip and male waist-to-shoulder ratios as determinants of romantic partner desirability. Journal of Social and Personal Relationships 23: 805–819.
• Carroll, Noe¨l. 1990. The philosophy of horror or paradoxes of the heart. New York: Routledge.
• Carroll, N. 2006. Aesthetic experience: A question of content. In Contemporary debates in aesthetics and the philosophy of art, ed. Matthew Kieran, 69–97. Malden: Blackwell Publishing.
• Costikyan, Greg. 2002. I have no words & I must design: Toward a critical vocabulary for computer games. http://www.digra.org/dl/db/05164.51146.pdf. Accessed 28 May 2010.
• Currie, Gregory. 1995. Image and mind: Film, philosophy, and cognitive science. Cambridge: Cambridge University Press.
• Damasio, Antonio. 2005. Descartes’ error: Emotion, reason, and the human brain. New York: Penguin.
• Decety, Jean, and Philip L. Jackson. 2004. The functional architecture of human empathy. Behavioral and Cognitive Neuroscience Reviews 3: 71–100.
• Dimberg, Ulf, Monica Thunberg, and Kurt Elmehed. 2000. Unconscious facial reactions to emotional facial expressions. Psychological Science. doi:10.1111/1467-9280.00221.
• Eimer, Martin, Amanda Holmes, and Francis P. McGlone. 2003. The role of spatial attention in the processing of facial expression: An ERP study of rapid brain responses to six basic emotions. Cognitive, Affective, & Behavioral Neuroscience 3: 97–110.
• Ekman, P. 1999a. Basic emotions. In Handbook of cognition and emotion, ed. Tim Dalgleish and  Mick Power, 45–60. Sussex: Wiley.
• Ekman, P. 1999b. Facial expression. In Handbook of cognition and emotion, ed. Tim Dalgleish and Mick Power, 301–320. New York: Wiley.
• Ekman, I. 2008. Psychologically motivated techniques for emotional sound in computer games. In Proceedings of the Audio Mostly Conference, 20–26. Audio Mostly archive. http://www.audiomostly.com/images/stories/proceeding08/proceedings_am08_low.pdf. Accessed 6 May 2010.
• Ekman, I., and P. Lankoski. 2009. Hair-raising entertainment: Emotions, sound, and structure in Silent Hill 2 and Fatal Frame. In Horror video games: Essays on the fusion of fear and play, ed. Bernard Perron. Jefferson: McFarland.
• Hjort, M., and S. Laver. 1997. Introduction. In Emotion and the art, ed. Mette Hjort and Sue Laver, 3–19. Oxford: Oxford University Press.
• Isbister, Katherine. 2006. Better game characters by design. Amsterdam: Morgan Kaufmann.
• Järvinen, Aki. 2008. Games without frontiers: Theories and methods for game studies and design. Tampere: Tampere University Press.
• Juul, Jesper. 2005. Half-real: Video games between real rules and fictional worlds. Cambridge: The MIT Press.
• Lamarque, P. 2004/1981. How can we fear and pity fictions? In Aesthetics and the philosophy of art, ed. Peter Lamarque and Stein H. Olsen, 328–336. Malden: Blackwell Publishing.
• Lankoski, Petri. 2010. Character-driven game design: A design approach and its foundations in character engagement. Aalto University: Helsinki.
• Lankoski, Petri. 2011. Player character engagement in computer games. Games and Culture 6: 291–311. doi:10.1177/1555412010391088.
• Moffat, David C., and Katharina Kiegler. 2006. Investigating the affects of music on emotions in games. In Proceeding of Audio Mostly. http://www.tii.se/sonic_prev/images/stories/amc06/amc_proceedings_low.pdf. Accessed 5 Feb 2008.
• Molholm, Sophie, Walter Ritter, Micah M. Murray, Daniel C. Javitt, Charlers E. Schroeder, and John J. Foxe. 2002. Multisensory auditory–visual interactions during early sensory processing in humans: A high density electrical mapping study. Cognitive Brain Research 14: 115–128.
• Morrison, India, and Tom Ziemke. 2005. Empathy with computer game characters: A cognitive neuroscience perspective. In: Proceedings of the Joint Symposium on Virtual Social Agents: AISB‘05, 73–79. AISB, UK.
• Nichols, Shaun. 2004. Imagining and believing: The promise of a single code. The Journal of Aesthetics and Art Critisism 62: 129–139.
• Oatley, Keith. 1992. Best laid schemes. Cambridge: Cambridge University Press.
• Pan, Xueni, and Mel Slater. 2007. A preliminary study of shy males interacting with a virtual character. In Presence 2007: The 10th Annual International Workshop on Presence. http:// www.cs.ucl.ac.uk/staff/S.Pan/PanandSlater.pdf. Accessed 28 May 2010.
• Paradiso, Sergio, Debra L. Johnson, Nancy C. Andreasen, Daniel S. O’Leary, G. Leonard Watkins, Laura L. Boles Ponto, and Richard D. Hitchwa. 1999. Cerebral blood flow changes associated with attribution of emotional valence to pleasant, unpleasant, and neutral visual stimuli in a PET study of normal subjects. The American Journal of Psychiatry 156: 1618–1629.
• Perron, B. 2009. The survival horror: The extended body genre. In Horror video games: Essays on the fusion of fear and play, ed. Bernard Perron, 121–143. Jefferson: McFarland & Company.
• Power, Mick, and Tim Dalgleish. 1997. Cognition and emotion: From order to disorder. Hove: Psychology Press Ltd.
• Ravaja, Niklas, Timo Saari, Mikko Salminen, Jari Laarni, and Kari Kallinen. 2006. Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology 8: 343–367.
• Reber, Rolf, Norbert Schwarz, and Piotr Winkielman. 2004. Processing fluency and aesthetic pleasure: Is beauty in the perceiver’s processing experience? Personality and Social Psychology Review 8: 364–382.
• Salen, Katie, and Eric Zimmerman. 2004. Rules of play: Game design fundamentals. Cambridge: The MIT Press.
• Sloboda, J.A., and P.N. Juslin. 2001. Psychological perspectives on music and emotion. In Music and emotion: Theory and research, ed. Patrik N. Juslin and John A. Sloboda, 71–104. Oxford: Oxford University Press.
• Smith, Murray. 1995. Engaging characters: Fiction, emotion, and the cinema. New York: Oxford University Press.
• Tavinor, Grant. 2009. The art of videogames. Chichester: Wiley-Blackwell.
• Walton, Kendall. 1990. Mimesis as make-believe: On foundations of the representational arts. Cambridge: Harvard University Press.
• Zehner, S.M., and S.D. Lipscomb. 2006. The role of music in video games. In Playing video games: Motives, responses, and consequences, ed. P. Vorderer and J. Bryant, 241–258. Mahwag: Lawrence Erlbaum Associates.

Games

• Eurocom. 2003. Buffy the Vampire Slayer: Chaos Bleeds.
• gameLab. 2005. Diner Dash.
• Ion Storm. 2002. Deus Ex. PS 2.
• Ion Storm. 2005. Thief Deadly Shadows.
• Level-5 Inc. 2002. Dark Chronicle.
• Looking Glass Studios. 2000. Thief II: The Metal Age.
• Maxis Software Inc. 1989. SimCity.
• Pazhitnov, Alexey. 1985. Tetris.
• Rockstar North. 2002. Grand Theft Auto: Vice City.
• SEGA Corporation. 2005. Super Monkey Ball Deluxe.
• Sony Computer Entertainment. 2002. Ico.
• Sucker Punch Productions LLC. 2005. Sly 3: Honor Among Thieves.
• Team ICO. 2006. Shadow of the Colossus.
• Team Silent. 2002. Silent Hill 2: Director’s Cut.
• Team Silent. 2003. Silent Hill 3.
• Ubisoft Entertainment. 2003. Tomb Raider: The Angel of Darkness.
• Ubisoft Montpellier Studios. 2003. Beyond Good & Evil.