Observations

(c) 2017  Petri Lankoski, v0.1 (a very drafty draft.)

The purpose of observational design in DGR is to understand how players interact with the game, how the system shapes the play and the player’s reactions to the game events. An observational design also means that there is someone else than the researcher playing. The researcher playing and recording the play is another type of study design (see analyzing previous designs). Also, if the focus is in the player-player interactions or relations in, for example in MMORPG ethnography or participatory observations might suit better for that than the observations described here. For ethnography to study MMORPG play, see Brown (2015).

Observations can be used in qualitative, correlational, quasi-experimental, and experimental to mixed methods research designs.

Data gathering

The goal of an observational design is to study game systems via observing gameplay. The basic set-up is let players play a game and observing them while they play. Recording the game session to in some way is advisable. For video recording, researchers should record the game screen(s) and the player(s) at the same time (see fig below). In addition, it is possible to use event information from game logs, server requests the game makes, or instrument the game to send or log the gameplay events of interest. Using logs or instrumenting the game requires is that the game is developed for research or the game can be modded to log events.

Figure. An example of a laboratory setup for data recording data

A variation of observations is observations with the think-aloud protocol. Note that the think-aloud protocol is hard for players when the game is demanding. In the think-aloud protocol, players are asked to explain why they are doing what they are currently doing. (An alternative to the think-aloud protocol is using the stimulated recall interviews. About stimulated recall, see Pitkänen, 2015, where after the game session, the players are showed specific game events and asked about what they were thinking in those situations and why they performed in that way, etc.).

Qualitative analysis

Observation data can be analyzed qualitative, quantitative, or mixed methods approach. However, in all cases, the data needs to be coded. (Note that there are ways to automate the coding of game events in some extend [cf., Marczak and Schott, 2015]) .

Below, coding analysis approach is described in more detail. Discriptions about other possible qualitative analysis methods are on section Qualitative data analysis.

Coding

There are two approaches for data coding: pre-set codes and emergent codes. In both coding approaches, researchers connect an event in the data with a code and time stamp so that the event is easy to find later. It is also important that one code represents a type of event and all that kinds of event are content with that code.

The codes names itself should be brief and descriptive. It is a good practice to have a more verbose description of what code means as well as some examples of that code.

First phase

Pre-set codes

The coding schemata are theory-based. As an example of a theory based coding schemata, Ravaja, et al. (2005) set-up codes for observing events in Super Monkey Ball () based on psychological theory: any events that are likely to provoke emotional responses. They ended up with following pre-set codes:

“Start”           Playing starts (the end of loading)
“End”             Playing ends (starts loading)
“Go!”             Level begins, “go!” is displayed (+sound)
“Small fall”      A small fall onto a platform below
“Seeing stars”    Hitting the platform so that the monkey sees stars
“Sparkles”        Going so fast that sparks fly
“Home stretch”    Finish line comes into view without obstacles, 
                  home stretch begins
“Goal post bump”  Bumping into a goal post (instead of finishing the 
                  race)
“Finish”          Finishing the race
“Time bonus”      Getting a time bonus (after finishing the race?)
“No time bonus”   Not getting a time bonus (after finishing the race?)
“Replay”          Replay starts
“Big fall”        Falling off the platform
“Switch”          Pressing a switch on the ground
“Banana”          Picking up a banana
“Banana bunch”    Picking up at least five bananas or one bunch of bananas
“Map zoom”        Zooming map
“Bump”            Bumping into a wall, bump, and so on 
(Ravaja, et al., 2005)

Emerging codes

This is a data-driven approach where codes emerge from inspecting data. These emergent codes describe the events in the data. The main rule is that researchers should not make the data fit the codes, but create codes that describe the data. Using emerging codes is a recursive process: after new codes are created, the data needs to be analyzed again using the new set of codes.

Table. An example of coded data from Watchdogs 2 (Ubisoft Montreal, 2016).

Player 1 (man, 29 years old)
Coded event     Timestamp        Notes
...
Lift Hack       10:10            Raising scissorlift
Selfie          10:29
Lift hack       10:30            Moving lift
Selfie          10:42            Selfie now with a graffiti on backfround
Jump            10:51            Jumping down from the lift
Damage          10:52            Landing from too high; receiving damage
Run             10:52
Hit             10:56            Hit by a bus, died
Game cont.      11:21
Run             11:21            Running on street
Jump            11:23            Jump over a car
Run             11:26
Attack          11:28            Striking a bystander
Run             11:28
Jump            11:34            Jumping over a car
Run             11:35
Climb           11:44            Climping a wall
Run             11:45
Pet             12:00            Petting a dog
Walk            12:01
Pet             12:06
Hack/Stealing   13:11
Run             13:11
Jump            13:24            Jumping over a fence
...

Of course, coding everything is not usually feasible as there are just too many events happening. The research question gives focus for coding and only what is relevant for the research question should be coded.

Mixed codes

This coding approach uses pre-set codes and emergent codes. For example, the player’s reactions to game events could be coded using Paul Ekman’s Facial Action Coding System (FACS) and game events relating to those emotional reactions can be coded using emergent codes.

Coding example

Second phase coding

In the second phase, initial codes are grouped together by creating categories from the interconnected codes.

In coding, researchers would be asking, for example, if there are context, conditions, consequences or actions that connect the coded events.

From the categories and connections created in the axial coding process, researchers build a generalized description what happens based on the data. That description can even be a model and hypothesis.

Back to Watchdog 2. As Watchdog 2 is a sandbox game that allows multiple different play styles and behaviors, research question about how players are using the game are actually playing the game is an interesting one. By coding multiple players actions we could end up something as follows. [NEED TO WORK THIS…]

  • Travel (players prefer different ways of travel)
    • Parkour
    • Racing
    • Walking
  • Violence
    • Random hostility
    • Combat in Quests
  • Quest play (players prefer different ways of travel)
    • Stealth / Marcus
    • Stealth / Remote control
  • Non-quest behaviour
    • Selfies
    • Parkour
    • Vehicle stunts

Validation

Interrater agreement…

Qualitative analysis example (with a real data)

Example coming here…

Quantitative analysis

Basics of quantitative approaches are described in section Quantitative approaches to study games. In this section, we give an example about quantitative analysis of observational data.

Mixed methods design

An example

XXX

Research plan

Aims of the study

The game

  • description of what parts will be used and how; difficulty level, …

Sampling

What data is gathered and how

  •  Video record players and gameplay. If players will be recorded, how many cameras is used…
  •  game logs, what is logged
  • should something blinded; if so, how

Data handling

  • how the data is stored and how long it is stored, data protection
  • destroying the data vs anonymising the data (video or voice recordings are hard to anonymise)

Analysis plan

  • Qualitative
  • Quantitative: correlation or experimental

Ethical considerations

Video and voice recordings that show participants and where you hear participants voices pose potential ethical and legal issues. The recordings can reveal who participants are and the recordings are hard to anonymise. The data can reveal personal and sensitive details about participants so it is important that the data is handled with care.

Principles

  • Participants are aware and accept how the recordings are used and how long the
  • Recordings should be anonymised or destroyed after the data has been analyzed and reported

Exercises

Data collection

Coding

References

  •  Braun and Clarke, 2006
  • Loland, J., Snow, A., Anderson, L. and Lofland, L.H., 2006. Analyzing social settings: A guide to qualitative observations and analysis, 4th ed. Bemont: Wadsworth Cengage Learning.
  • Ravaja, Saari, Laarni, Järvinen, 2005. The Psychophysiology of Video Gaming: Phasic Emotional Responses to Game Events. In: DiGRA 2005 conference proceedings.
  • Saldaña, J., 2016. The coding manual for qualitative researchers. Los Angeles: SAGE.