Evaluating Construct Convergence: A Canonical Correlation Analysis of Diagnostic Classification Model-Based SJT and IPIP-NEO-PI-120

¹ Faculty of Psychology, Gadjah Mada University, Yogyakarta, Indonesia
² Faculty of Psychology, Sanata Dharma University, Yogyakarta, Indonesia
³ Department of Psychology, Indonesia University of Education, Bandung, West Java, Indonesia

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

The Five-Factor Model (FFM) is the most widely used framework for understanding personality, typically assessed through self-report inventories such as the IPIP-NEO-120. Although these tools show strong psychometric properties, reliance on self-reports introduces vulnerabilities to bias, including social desirability, response distortion, and faking, particularly in high-stakes contexts like personnel selection. To address such limitations, researchers have increasingly explored behavior-oriented measures, notably Situational Judgment Tests (SJTs). When analyzed using Cognitive Diagnostic Models (CDMs), SJTs can yield multidimensional mastery profiles that offer a nuanced representation of personality. However, limited evidence exists regarding the convergence between CDM-based SJT profiles and traditional self-reports. This study applied Robust Canonical Correlation Analysis (RCCA) with a Spearman rank correlation matrix to examine structural alignment between CDM-based SJT scores and IPIP-NEO-120 domain scores in a sample of 289 participants. RCCA was chosen to handle common violations of multivariate assumptions, such as non-normality and outliers. Results showed a moderate canonical correlation in the first root, supporting convergent validity while highlighting distinct trait representations across methods. Subsequent roots yielded weaker and less stable associations. Overall, findings indicate that RCCA is a viable approach for evaluating cross-method validity and emphasize the complementary insights of situational and self-report approaches in personality assessment.