From Play to Validity: A Design–Psychometric–Cognitive Framework for Evaluating Game-based Personality Assessment

¹ Faculty of Psychology, Gadjah Mada University, Yogyakarta, Indonesia
² Faculty of Psychology, Sanata Dharma University, Yogyakarta, Indonesia
³ Faculty of Art and Design, Institute of Technology Bandung, Bandung, Indonesia

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Game-Based Assessments (GBAs) offer innovative approaches to measuring psychological constructs by embedding tasks within interactive environments. Despite their potential, concerns remain about validity and whether gameplay reflects stable traits rather than situational or design-driven factors. This study aimed to examine the construct validity of a GBA designed to measure Conscientiousness by integrating cognitive interviewing, Item Response Theory, and design evaluation. Ten participants completed the GBA and participated in follow-up interviews guided by Collins’ framework. The study addressed how gameplay tasks represented Conscientiousness and which design features affected players’ cognitive responses. Findings showed that early stages (1–2) demonstrated strong alignment, clear instructions, understandable goals, and items with adequate fit and discrimination, eliciting structured, trait-consistent behaviors. Later stages (3–6) exhibited weaker psychometric performance, divided attention, and confusion from visual similarity. Overall, results highlight the need to align psychometric rigor, cognitive clarity, and design simplicity in developing valid GBAs.