Content Quality of Lesson Scenarios in Moscow E-School

. This work is aimed at assessment of content quality of lesson scenarios in Moscow e-school, comparison of expert data with the features of application of these scenarios in order to reveal strong points and deficiencies in content of lesson scenarios. The considered lesson scenarios are characterized by high quality of material component of educational content. The lesson scenarios substantiate forms and types of control of educational results. Methodological support of the lessons, differentiation and individualization of the content, existence of nonstandard situations stimulating personal participation of students, instructions for a student have been manifested least in the lesson scenarios. High expert quality appraisals of lesson scenarios mostly do not coincide with the popularity of lesson scenarios among the users.


Introduction
Moscow e-school (MES) is one of the largest educational systems in the Russian Federation. The MES library is a unique storage of educational materials, available to any student, teacher, or parents. In personal cabinets of MES teachers (as of May 30, 2020) there were 1,413,646 lesson scenarios, 154,311 applications, 167,227 tests, 45,321 e-schoolbooks, several millions of atomic contents. More than 44 thousand lesson scenarios were moderated and are available for all users in the library. Schleicher believes that MES consolidated parts of the system and provided scaling up, thus allowing various components of the system to progress in one direction. The open content served as the center of giant teachers' community, which is related with natural wish of people to work together, to cooperate, and to gain recognition for their work: it is important that the teachers understand significance of their activities, and the students -of their education [1].
The studies were based on data about operation of users with MES lesson scenarios. These data were obtained by uploading from analytical system and revealed by expert analysis of lesson scenarios. Herewith, it was taken into account that the activity of teachers in the MES system was unrelated to a particular school: teachers were free to use scenarios developed by the authors from other schools, where application of ready scenarios was encouraged and promoted [2].
The studies covered 18 subjects of secondary education.

Methods
Expert card was developed for the studies. Its development was based on the data acquired upon analysis of requested lesson scenarios deposited in MES library [3]. The expert card includes quality appraisal of scenarios content regarding four fields: material, procedural, motivation and value, ideal components. Each field of appraisals is comprised of four indicators (see Table 1). Total information and data on the use of lesson scenarios were obtained automatically from MES IDVP DATA analytical system. They are as follows: year of creation of lesson scenario, its rating, number of copies taken from this lesson scenario, amount of addition to selection, number of runnings the lesson scenario, whether this lesson scenario is a copy, whether it has a grant. The data on reviews of lesson scenarios were not analyzed since reviewing assumes just user session to preview the lesson scenario. All lesson scenarios selected for analysis were moderated and published on open access. No common ranges were introduced for data obtained automatically from analytic system since the number of usages (running, copying, addition to selection, etc.) depended on the number of hours of educational subject in school program, number of teachers, etc. For instance, comparison of usage of lesson scenarios for the Russian language and music is not correct.
273 lesson scenarios for 18 subjects of secondary education were selected for expert appraisal: the Russian language, literature, foreign languages (English, German, French), mathematics, algebra, geometry, informatics, biology, chemistry, physics, geography, history, society science, music, graphic arts, physical training. Fifteen lesson scenarios were selected automatically for each subject (five with the highest number of running, five with the average number of running, and five with the zero running), except for graphic arts and physical training. Fourteen of 19 lesson scenarios were selected, respectively. Each lesson scenario was appraised by two experts, except for mathematics, algebra, and geometry, which were appraised by one expert. 546 expert appraisals were available for analysis.
Expert appraisal was carried out using digital expert card. Lesson scenarios and information about their usage by teachers were distributed among experts automatically.

Results
While referring lesson scenario to one of the typological groups, the experts generally selected the following variants: "by main didactic purpose" and "by activity approach". Diversions between the expert appraisals were more often related with the fact, in terms of which typological group an expert considered the lesson scenario. In addition, five top most popular selections of lesson type by experts could be determined: lesson introducing into a new material, lesson of discovery of new knowledge, lesson of gaining new skills and habits, combined lesson, lesson of knowledge systematization (general methodological orientation), lesson of applying knowledge and skills.
Let us present the results of data analysis of overall array of expert appraisals of content of lesson scenarios. The expert detected existence (yes) or absence (no) of indicator.

Discussion
Key trend of MES development is improvement of quality of educational materials, refusal of "slag" content of education. The following results were obtained. Material component was the first level of lesson content appraised by the experts. The experts believe that in 94.0% of cases, the texts, audio-, video-, and illustrative materials, and in 91.2% of cases, the test and interactive assignments in lesson scenarios corresponded to the content. In 13.6% of cases, the scenarios contained actual errors. This should be excluded, since all lesson scenarios, presented for appraisal, were moderated. In 19.4% of cases, the experts stated lesson overloading with information. However, most lesson scenarios corresponded to all requirements to the quality of material component of lesson scenarios.
The situation concerning quality of procedural component of lesson scenario -learning and communicative actions, a system of exercises -was somewhat different. In 59.5% of cases, the experts determined interaction with objective and/or digital medium at the lesson. Differentiation and individualization of content were detected only in 21.8% of cases. In 39.5% of cases, the experts determined existence of assignments for students' work in minor groups, and in 37.1% -substantiation of homework. Only in one third of the considered lesson scenarios it was possible to mention correspondence of the procedural component of content to the presented requirements. This result is stipulated by domination of explanatory and illustrative method during teaching school subjects.
While analyzing motivation and value component of educational content in lesson scenarios, the experts mentioned that in 79.6% of cases, the forms and types of control of educational results were motivated. The numbers of expert detections of existence or absence of reflection and/or self-appraisal as a stage in lesson scenarios were nearly the same (49% and 51%). Existence of assignments and/or situations for formation of value attitude to culture was detected in 42.1% of expert appraisals. 60.9% of expert appraisals indicated the absence of personal experience of students in lesson scenarios. These results are stipulated by different specifics of subject content and evidence low level of usage of methods to involve students into the studied issues.
83.2% of expert detections of ideal component of education content (problems, issues, problematic situations) indicated at clear formulation of lesson subject. In 65.3% of cases, the experts detected the absence of problematic situations. The expert appraisals were subdivided into nearly equal portions with regard to the indicator titled Search pattern of questions and assignments. Existence of nonstandard situations stimulating personal involvement of students was detected only in 25.3% of expert appraisals. These results in the absence of mechanisms for creation of condition for immersion of students into active discussion of various problems in nearly 75% of lesson scenarios, and in low orientation of existing education content on formation of new educational results.
Comparison of data on user operation with lesson scenarios obtained by data downloading from MES system and expert appraisals in order to determine the most qualitative and demanded lesson scenarios in most cases (except for biology and physical training) revealed mismatching between the expert appraisals and user operation with lesson scenarios. More often higher expert appraisals were given to the lesson scenarios which were rarely used (run) during learning. This indirectly evidences that teachers more frequently apply conventional lesson scenarios, not assuming arrangement of various activities of students at the lessons.
In addition, there existed cases when lesson scenarios were added to selections or became copies numerous times (50 times and more) and were not used at all during a lesson.
Similar issues were met in numerous network communities of developers of learning content [4,5].
Lesson scenarios and other learning processes are actively accumulated and repeatedly used in the whole world. Classification systems of educational practices are becoming international standards. The international standard of description of learning processes IMS LD makes it possible to develop scenarios of learning processes, which can be repeatedly applied by other teachers. Description of learning scenarios is based on the work by Schank and Abelson [6], where a scenario is presented by the sequence of actions of characters with objects. At present, this sequence is generally written in the form of diagram [7,8]. In international community the design of learning scenarios is related with the standard of learning scenarios description in IMS LD [9]. In Russian studies, in the scope of scenario a script of presentation sequence of learning materials is executed [10,11]. Upon nonformal approach, the attention is paid to the actions of student [12]. The most popular repository of learning scenarios was CloudWorks community [13][14][15], where teacher could exchange, compare and analyze lesson scenarios. Unfortunately, in 2019 the community closed the opportunity of load new scenarios, it operates only as storage. This could be possibly attributed to stringent rules of description of lesson scenarios and problematic selection between storage of high-quality content and maintaining high interest of participants to be authors of this content.

Conclusion
In the course of expert appraisal of lesson scenarios, the expert card and digital tools of appraisal of lesson scenarios were approbated. In the future it will be possible to perform expert appraisal on the basis of one typological group and to determine indicator weight for each subject. Expert card can be improved with consideration for the conditions of widescale conversion of schools to distance learning. In addition, at this stage it was impossible to perform systematic correlation analysis of data on the user actions with lesson scenarios in MES and their expert appraisals. In the light of digital transformation of educational system, development of tools of intelligent data analysis, a promising step is development of algorithms of content quality analysis of lesson scenarios. However, this is the subject of separate study.