Design skills assessment in teaching 3D modeling

The article discusses the design skills assessment method, which was developed and used to teach basics of three-dimensional modeling. This method is necessary because the creation of a virtual 3D model usually does not include the design phase as a separate phase with own specifications and design documentation and all design activities are performed in mind without any preliminary sketches. The offered method allows assessing the quality of the student's mental design operations based on the analysis of the 3D modeling process tracking. The student's activity in the 3D modeling environment was captured using special software. The obtained data allow to identify the essential parameters of the students' actions (correctness, optimality, etc.), which were used to quantify their design skills. The construct validity of the proposed method and its statistical significance of the assessment were confirmed.


Introduction
Designing a three-dimensional model is an essential component of its development, since it provides the efficiency of the method of its manufacture, as well as the quality of the result obtained. Therefore, when teaching the basics of three-dimensional modeling, special attention should be given to the development and evaluation of design abilities [1]. The existing methods for design skills assessment are developed primarily for the training of specialists in the field of industrial design and cannot be directly transferred to the field of three-dimensional modeling. These methods can represent design skills in the form of a set of components, such as Analytical Skills, Ability to Use Design Tools, etc. [2], or they can be based on identifying and measuring design skills that are obtained from observations of the design tasks implementation [3].
The inability to use these methods to assess design skills in the field of threedimensional modeling directly is due not only to the technological differences in the production of industrial and virtual 3d objects, but also to the various requirements imposed on the final products of these industries. In addition, 3d design phase is usually performed in a free format without complying with any rules and standards for the design documentation. Sketch is the most adequate form of the design stage/phase in the field of three-dimensional modeling. Since this form is an effective tool for visual thinking, mastering this form of design execution is an important part of professional training in the field of engineering design [4]. But unfortunately, most often, developers of threedimensional models do not even consider the design process itself as a separate phase. Skipping this phase is typical situation, especially for some students, who do not understand why they need it at all, or for the professionals who completely perform design process in their minds.
This design approach makes it impossible to assess how well the geometric model of the form is thought out, and whether the process of modeling itself is optimal. The only parameter of design abilities, the assessment methodology of which is currently fully developed, is the parameter of the geometric grid of the three-dimensional model optimality. There are automated ways to do this assessment [5].
These circumstances led to the need to develop a design skills assessment method, which would allow to assess the presence and quality of the student's mental design operations.

Theoretical Framework
The theoretical basis that provides the construct validity of the proposed method is the theory stage-by-stage formation of mental actions by P.Ya. Galperin [6]. According to this theory, students' design actions can be presented as an orienting activity, because it gives the opportunity to adapt to specific conditions in order to achieve a given goal. In other words, orienting activity ensures the fulfillment of specific requirements for the final product (three-dimensional model in our case), as well as one or another degree of correctness of the actions necessary to obtain this product. Depending on the degree of development of the orientation activity, the qualitative and quantitative characteristics of the operations, performed to achieve the goal, change. Undeveloped (incomplete) forms of orientation lead to spontaneous actions and to a large number of errors, that often affects the quality of the final result. In addition, at some stage of development, some operations may move from the external material form into the internal mental one. At the same time, the qualitative and quantitative characteristics of mental actions will fundamentally depend on their earlier external form [7].
Applying the described theory to the process of three-dimensional models developing, we can state the following:  the documented process of a 3d model design development (for example, in the form of a sketch) is an external form of orientation;  over time, the external form of the orientation can be internalized;  the way of the 3d model development, its qualitative and quantitative characteristics will depend on this process. Therefore, if students have learned to design three-dimensional models in the form of sketches that ensure the correctness and optimality of the development of the final product, then their internal design actions will be equally correct and optimal. Obviously, depending on students' design skills level, the characteristics of their operating structures in the instrumental environment will change. Registration and analysis of these characteristics were used as the basis for the practical implementation of the proposed method.
Thus, the general idea of the method is as follows: at the first stage, students' design skills are assessed directly by the material results of their project activities, namely according to:  projects made in the form of sketches;  the description of the process of modeling technology selection;  the model, produced according to the developed project.
At the next stage, pupils are invited to perform a 3d model at once, implying that they will perform all the project activities in their mind. All pupils' external activities are tracked.
The obtained data allows both to reveal the significant factors of external operations that affect design abilities, as well as to build a regression model of these abilities.

Implementation
The experiment was carried out in the pre-university training system of the ITMO University [8] during the "Basic 3D modeling course". The experiment involved 112 school children aged 12-14.
In accordance with the proposed theoretical approach, at the first stage, pupils were offered to design a 3d model, which could be created in different ways. The project was carried out by the pupils in the form of a sketch, where they had to describe possible methods of modeling and choose the best one (Fig. 1a). These data reflect the process of finding design solutions directly and also its quality, as well as the level of development of divergent (count of options) and convergent (ability to choose the best option) thinking of pupils. Comparison of the project and the model developed on this project (Fig. 1b) allowed assessing the ability of pupils to implement the conceived model. At the second stage, the pupils were asked to develop a model, the project of which they were able to implement in their minds (Fig. 2). All operations in the 3ds Max tool environment, performed by pupils during the test model implementation, were recorded using the UVScreenCamera video capture program. The resulting video (a frame of the video is shown in Fig. 3) was later used to determine the factors of instrumental activity of students. One of the issues that had to be solved in the process of formation of a design abilities model was the issue of determining the variety of factors of pupils' instrumental activity, on which the level of design skills would depend. According to Galperin [9], the most complete properties of actions are described by the following factors: a) correctness of the way to perform actions (correctness of the actions); b) correctness of the 3d model; c) time of performing actions; d) optimality of operational structure (optimality of actions); e) criticality.
In the context of the three-dimensional modeling process, recorded using video, these factors were considered as follows: a) the factor of the correctness of the actions was counted as the ratio of the count of correct actions to the total count of actions, taking into account the correctness factor determined for each action (in percent); b) the factor of the correctness of the 3d model was defined as the degree of conformity (in percent) of geometric shapes of the sample and the model created (compliance of structural units and proportions was taken into account); c) the time of performing actions was fixed using a timeline; d) the factor of optimality of actions was counted as the ratio of the count of actions in the optimal modeling process (a constant count) to the total count of actions performed to create the model (in percent). The only factor that could not be determined from the video was the criticality factor. Therefore, upon completion of the modeling process, pupils were asked to evaluate both the quality of the product developed and the correctness of the operational structure of the actions performed. The criticality factor (e) was counted as the ratio of the count of pupil's errors mentioned to the total count of errors committed by him.

Results and Discussion
In result of the regression analysis, a procedure of the number of factors minimization was carried out, the unstandardized and standardized coefficients were determined. The factor of the resulting model correctness and the time factor were excluded at the stage of the search for multicollinear factors. As a result, the following factors turned out to be the most significant:  correctness of the actions;  criticality;  optimality of actions. Table 1 shows the main parameters of the regression model. Acceptability of the model was confirmed using the p-value (<0.05) and R square (>0.4) criteria. As we can see from the table, the most statistically significant factor is the factor of correctness of actions. The second most important factor is the criticality factor, which, unfortunately, is very difficult to determine from the video capture of instrumental actions. However, the fact that criticality is required for the development of actions [9], gives hope for measuring this parameter by comparing actions at an earlier and later stages of training.
In conclusion, we can say that the proposed method can be applied to assess design skills in those areas of computer technology where it is possible to record the student's activity in the instrumental environment.
The developed model for design skills assessment is planned to be used for the undergraduate students, studying "Computer graphics and design" course in the ITMO University.