Measuring Perceived Challenge in Digital Games: Development & Validation of the Challenge Originating from Recent Gameplay Interaction Scale (CORGIS)

In short

Players can find a game challenging in more ways than needing quick reflexes. This paper presents CORGIS, the Challenge Originating from Recent Gameplay Interaction Scale: a questionnaire that measures how challenging players found the game they just played. Its items came from a review of games user research, game design and game AI literature, and from interviews with researchers and players.

In games

What the four kinds of challenge feel like

In the paper

Civilization

Strategy series

Cognitive challenge: planning many turns ahead and juggling cities, research and diplomacy at once.
Our example

Celeste

2018

Performative challenge: precise, split-second jumps that you get right only with practice.
In the paper

XCOM

Strategy series

Emotional challenge: with permanent death, “when your character dies – they die, and you feel it”, as one player put it.
In the paper

Life is Strange

2015

Decision-making challenge: choices with lasting consequences, which left players wondering what would have happened had they chosen differently.

How CORGIS was developed

Literature review: Operationalising challenge as player experience and identifying the factors that might lead to challenging experiences
Interviews: Semi-structured interviews with four players and five games user researchers (N = 9)
Item generation: A 60-item pool based on the themes from the literature review and interviews
Exploratory factor analysis: An online survey about players’ most recent gaming experience: 394 included respondents (432 took part)
Confirmatory factor analysis: A second survey about a recent game-playing session: 987 included respondents (1001 took part)
Criterion validity: Data from players of three games held out from the factor analysis (N = 148)

Challenge profiles of three games

The games “have quite different styles”, so the authors formulated hypotheses for each kind of challenge before the analysis. Mean scores on each subscale (1–7):

Life is Strange n = 36

Cognitive: 3.59
Emotional: 6.21
Performative: 3.07
Decision-making: 6.28

Monster Hunter World n = 47

Cognitive: 5.62
Emotional: 3.43
Performative: 6.24
Decision-making: 3.33

XCOM n = 65

Cognitive: 5.97
Emotional: 4.60
Performative: 2.67
Decision-making: 6.12

Data from Table 8 of the paper (all differences p < .001). Yellow marks scores of 5.5 or above.

In players’ words

From the interviews behind the questionnaire.

The choice is you kill the demon which would solve the problem, but by doing so you would kill the child.
P1, on Dragon Age: Origins
If you play as what you consider a morally good character, and you consider yourself to be a morally good person, [the personalities and decisions] become intermingled quite quickly.
P5, on Skyrim

Key findings

Perceived challenge has four components: performative, emotional, cognitive and decision-making challenge.
The four-component structure emerged from an exploratory factor analysis with 394 players.
A validation study with nearly 1,000 players supported the scale's construct validity.

CORGIS · 30 items · 4 subscales

The questionnaire

Challenge Originating from Recent Gameplay Interaction Scale (CORGIS). Measures how challenging players found the game they have just played, across four kinds of challenge. Please cite the paper when you use it.

Download CSV Download text Qualtrics import

What do these give me?

Copy items: just the statements, one per line, ready to paste into any survey tool.
Copy everything or Download text: the full questionnaire with instructions, scale, scoring and citation.
Download CSV: one row per item with its ID, subscale and whether it is reverse-scored. Opens in Excel, R or Python and helps with scoring.
Qualtrics import: a file in Qualtrics' Advanced Format. In Qualtrics, choose Tools → Import/Export → Import survey. The items arrive as one matrix question; turn on statement randomisation if you want random order.

How to use it

Give the questionnaire immediately after the game session has finished.
Ideally, present the items in random order and without naming the four challenge types to players.
Use an agreement Likert scale: 5 points for a shorter survey, or 7 points for more precision.
All items are compulsory.

Instructions for players

Please indicate to what extent you agree or disagree with each of the following statements after you finished playing the game.

Response scale

1Strongly disagree
2Disagree
3Slightly disagree
4Neither agree nor disagree
5Slightly agree
6Agree
7Strongly agree

Items

Cognitive challenge 11 items

Arising from the need for planning ahead, memorisation, effort, preparation and multi-tasking.

CC1Succeeding in the game required much planning
CC2I had to memorise a lot of different things when playing the game
CC3I had to think several steps ahead when playing the game
CC4I had to prepare for the things that the game threw at me
CC5Playing the game requires great effort
CC6I felt challenged when playing the game
CC7I had lots of different things to think about at once in the game
CC8The game made me manage several tasks at the same time
CC9I had to constantly keep track of what was going on in the game
CC10I had to think actively when playing the game
CC11Playing the game required me to do my best

Emotional challenge 9 items

Arising from the emotions evoked in the player, which might also have implications for things they thought about outside of the game.

EC1This game is more than just a game to me
EC2The things that happened in the game made me sad
EC3I invested much thought into the game
EC4I felt a sense of responsibility for characters and events in the game
EC5The game made me think about real life issues
EC6Playing the game was stimulating
EC7I felt a sense of suspense when playing the game
EC8The game had moral dilemmas in it where the choice was not obvious
EC9The game involved making moral choices that I didn’t agree with

Performative challenge 5 items

Arising from the game requiring rapid and accurate action from the player.

PC1I had to react quickly when playing the game
PC2I had to act quickly when playing the game
PC3Thinking fast was an important part of the game
PC4Quickly responding to things that I saw was an important part of the game
PC5I had to make snap decisions when playing the game

Decision-making challenge 5 items

Arising from having to make choices that were difficult or could lead to regrettable outcomes.

DMC1There were some decisions in the game that I regretted
DMC2I wonder how different the outcome in the game would be had I chosen a different option
DMC3I had to make difficult choices in the game
DMC4I had to think about possible alternatives for my actions in the game
DMC5The game made me think hard about my decisions

Scoring

All items are positively scored; there are no reverse-scored items.
To score each challenge type, take the average of the responses to the items that belong to it. This gives four subscale scores; higher means more of that kind of challenge.
The published appendix also describes summing the items. Within a subscale, sums and means order players the same way, but means are easier to compare across subscales of different lengths.

Read and cite

PDF (accepted version) Publisher's version (DOI)

APA citation

Denisova, A., Cairns, P., Guckelsberger, C., & Zendle, D. (2020). Measuring Perceived Challenge in Digital Games: Development & Validation of the Challenge Originating from Recent Gameplay Interaction Scale (CORGIS). International Journal of Human-Computer Studies, 137, 102383.

Line of work

Builds on · 2017Challenge in Digital Games: Towards Developing a Measurement Tool
This paper · 2020Measuring Perceived Challenge in Digital Games: Development & Validation of the Challenge Originating from Recent Gameplay Interaction Scale (CORGIS)
Followed by · 2023Comparing Measures of Perceived Challenge and Demand in Video Games: Exploring the Conceptual Dimensions of CORGIS and VGDS