Predicting Personality Traits by Student Learning Behaviors on Blackboard Systems

Personality has been demonstrated as influential factors in technology-enhanced learning. The collection of personality is always a challenge. Human efforts are usually required in the user surveys which is the most common and popular way to collect the personality traits. Predicting personality traits, as a result, becomes one of the research directions. Some researchers consider these personality traits as labels in the classifications, while some others consider them as numeric variables in the regressions. In this paper, we made our attempt to predict the students’ personality traits from their learning behaviors on the Blackboard system. More specifically, we tried both the classification and regression models, and evaluate them based on the same standards. Our initial experimental results discover the insights about these predictive models.


Introductions and Related Work
Recommender systems is an effective solution to alleviate the problem of information overload and assist decision making. It has also been introduced to educations as one of the technology-enhanced learning techniques. For example, recommender systems have been built to suggest books for K-12 users [1], recommend after-school programs [2], or help select appropriate course projects in class [3,4]. In recent years, human factors, such as emotions, trust and personality, have been recognized as influential factors in the recommender systems. Particularly, the personality information has been demonstrated as influential factors in decision makings, especially in the area of educational learning. For example, Komarraju, et al. [5] identified the impact of personality on the academic achievements in GPA. Personality traits have been incorporated into traditional recommender systems [6][7][8], while some research distinguished user roles based on the personality traits in order to further improve the quality of group recommendations [9][10][11].
However, the collection of personality traits is always a challenge. One of the most popular ways is collecting personality information from user surveys. Personality can be measured by the personality traits. Several frameworks were developed to collect the personality traits. One of the most popular frameworks is the the Big Five Factor (Big5) [12] framework, in which the personality traits can be described by five dimensions [5]: Openness is reflected in a strong intellectual curiosity and a preference for novelty and variety. Conscientiousness is exemplified by being disciplined, organized, and achievement-oriented. Extraversion is displayed through a higher degree of sociability, assertiveness, and talkativeness. Agreeableness refers * e-mail: yong.zheng@iit.edu to being helpful, cooperative and sympathetic towards others. Neuroticism indicates the degree of emotional stability, impulse control, and anxiety. To collect the Big5 traits, survey is usually adopted, such as the Ten-Item Personality Inventory (TIPI) [13] questionnaire. These methods that rely on the user surveys usually require additional human efforts, not to mention that users' subjective information may not be reliable.
Predicting personality traits, as a result, becomes one of the alternative solutions. The predictions can be achieved by regressions or classifications. On one hand, the personality traits can be considered as numerical and continuous variables, so that the predictive models can be applied for predictions, in which the performance can be accessed by error-based metrics, such as mean absolute error. On the other hand, these personality traits can be treated as labels in the classification process. Ferwerda et al. demonstrate that visual and content features from Instagram pictures can be used to predict personality [14]. Kosinski et al. predicted the personality traits by using the user-likes on Facebook [15]. Chittaranjan et al. tried to infer the personality traits by from the usage data in the mobile phones (e.g., call and SMS logs) [16]. However, there are no existing research that compare the performance of these two categories of the methods. In this paper, we made our attempt to predict the students' personality traits from their learning behaviors on the Blackboard system 1 by using both classification and regression models. Our initial experimental results help discover the insights about the data and the effectiveness of the applied models.

Data Set
We collected our own data [17] for the purpose of experiments. First of all, we assigned a questionnaire to the graduate students in which they can fill in the demographic information, including age, gender, nationality, marriage status. In addition, we represented personality traits in the Big5 framework, and collected student personality traits by using the popular TIPI questions in the questionnaire. The demographic information are obviously not enough to predict students' personality traits. We additionally collected their learning behaviors on the Blackboard system. More specifically, instructors used Blackboard system to release assignments, and students can submit their homework to the Blackboard system. For each assignment, we are able to obtain three pieces of information -whether it is a late submission (i.e., binary variable), how many attempts were made for this assignment (i.e., numerical variable), and the SafeAssign score. SafeAssign is one of the components in the Blackboard system, and it compares submitted assignments against a set of academic papers or resources to identify areas of overlap between the submitted assignments and existing works. We collected the data from two courses over three semesters (i.e., 2018 Spring and Fall, and 2019 Spring semester), while there were around 10 to 11 assignments per course per semester.
We organize these three learning behavior variables (i.e., late, attempts, SafeAssign) in four blocks -by year (2018 or 2019), by semester (Fall or Spring), by year (2018 or 2019), and the overall block in which we merge all of the data associated with all assignments. For each block of the data, we calculate the descriptive statistics of the three variables for each student, including the mean, standard deviation, and skewness. It results in 63 new features, in addition to the 4 features from the demographic information. Finally, we obtained our data for experimental purpose. It contains the data for 130 students. For each student, there are 67 features in total associated with students' personality traits in five dimensions (i.e., Big5 framework). The values in the personality traits scale from 1 to 7 which describe the degree of the characteristics in the personality traits.

Methodologies and Results
As mentioned before, the personality traits can be treated as either categorical or numerical variables. Accordingly, we can use classification or regression models for the purpose of predictions respectively. More specifically, we can use multi-label classifications (MLC) to predict the personality traits, when we treat all of these five dimensions (i.e., personality traits in the Big5 framework) as categorical labels. In our experiment, we tried the mainstream MLC methodologies, including binary relevance [18], classification chains (CC) [19], label powerset methods [20]. CC is shown as the best multi-label classification method. We only present the results based on CC in this paper. Note that CC just defines a workflow to perform multi-label classifications. Actually we can adopt any traditional classifiers. In our experiments, we tried random forest (RF) [21], support vector machines (SVM) [22] and neural networks (NN) [23]. Alternatively, each dimension in the personality traits can be treated as a numerical variable. We tried the multiple linear regressions (MLR), support vector regressions (SVR) [24] and regression trees (RTree) [25].
Regarding the evaluations, we selected 10-fold cross validation due to the limited size of our data. In addition, we'd like to measure the performance on the same standards. More specifically, we measure the accuracy of each dimension in the personality traits when we apply MLC algorithms. Moreover, we also calculate the root mean squared error (RMSE) by converting the predicted ETLTC2020 SHS Web of Conferences 77, 0100 (2020) https://doi.org/10.1051/ onf /20 shsc 207701003 3 labels as the numerical score. Accordingly, we first measure RMSE when we utilize the regression models. Afterwards, we rounded the predicted values to integers and additionally measure accuracy for each dimension. By this way, we can present the accuracy and MAE for both the classification and regression models.
We present our results in Table 1, Figure 1 and Figure 2. More specifically, we present the best results of MLC and regressions in Table 1 only. The best model for MLC is using classification chain as the MLC workflow with neural networks as the classifier. The best regression model is using support vector regressions. Table 1 only present the results based on these best models. By contrast, we compare the performance by using different classifiers and regression models in Figure 1 and 2. Figure  1 presents the classification accuracy by using three classifiers (i.e., RF, SVM and NN), while Figure 2 describes the RMSE results by using three regression models (MLR, SVR and RTree).
We can observe that the optimal performance varies from dimensions to dimensions in Table 1. The accuracies are relatively higher and the RMSE values are lower for openness and conscientiousness, while the predictions are not that well in other dimensions. In terms of the accuracy, MLC algorithms outperform the regression models, while the regression models beat the MLC in RMSE. It maybe because that the predictions by MLC have no decimals, which results in higher prediction errors in RMSE. Similarly, the rounded predicted values by the regression models also lead to accuracy loss. By using principal component analysis (PCA) [26] for feature reductions, the accuracy and RMSE can be improved a little bit in the process of MLC. Even so, there are no clear winners between the MLC and regression models. It depends on which evaluation metric (accuracy or RMSE) is selected as the key criterion.
The results in Figure 1 and 2 are much more consistent. In Figure 1, we can observe that the best classifier is neural networks. The classification results are better in all the five dimensions in comparison with the accuracies by using other classifiers. In Figure 2, we can observe that SVR is the best predictive model. However, there are no significant differences between the performance between SVR and RTree in the dimension "conscientiousness".
The prediction accuracy is relatively low, which is consistent with the existing work [16]. The major limitation in this initial experiment is the size and sparsity of the dataset. First of all, the data is very small. In addition, we only obtained student behaviors in two courses. It is not guaranteed that each student will take all of these two courses, which results in the problem of sparsity.
We can also discover more insights from the best performing models. In all of the regression models, the age groups and nationality are the two most common and influential variables. They appear in most of the regression models. Other variables, such as the ones associated with descriptive statistics are only influential in predicting selected dimensions in the personality traits, including openness and extraversion. By contrast, these descriptive statistics from the features associated with students' learning behaviors are more important in the MLC models. We can observe that PCA produced 14 components as the transformed features, while each component is a linear representation of the original features. By observing the top principle components which capture the largest variance of the data, we can observe that the weights of different descriptive statistics features are much higher than the features associated with user demographic information. However, the top influential features may vary from components to components.

Conclusions and Future Work
In this paper, we exploit and summarize two possible ways (i.e., classification and regression models) to predict the personality traits. Our experimental results reveal that there are no clear winners, and the best choice depends on the evaluation metrics (e.g., accuracy or prediction errors). We identify that neural network is the best classifier in the MLC task when we view personality traits as categorical variables., while support vector regression becomes the optimal regression model if we consider the personality traits as numerical variables. Both features associated with demographic information and learning behaviors are important in MLC and regression models. However, the features based on learning behaviors may be only influential in selected personality traits. We plan to collect more data, and apply more complicated predictive models (such as the deep learning techniques [27]) in our future work to further improve the predictive performance on these personality traits.