Approaches to Assessing the Performance of Professional Expert Communities

. The article proposes the authors’ approach to assessing professional expert communities (PECs), based on the creation of a criteria scale and indicators of community activities. Using the comparative case study method, an analysis of the successful experience of 18 professional communities selected from an array of communities actively represented in cooperation with the institutes of the National Qualifications System is carried out. The content of the research is based on the use of the following research methods: content analysis, interviews with representatives of professional expert communities, case studies. It was found that the most successful communities, leading in the rating according to the results of the assessment, are the most consistent with indicators related to social criteria, and the least - to the material and technical, or economic. In addition, these PECs have a particular impact on socio-economic sustainable development from the viewpoint of the internal and external environment.


Introduction
A whole layer of works by Russian and foreign scientists is devoted to the study of the role of professional communities that affect the socio-economic dynamics in the regions, the country, both through inclusion in the innovation process and through participation in management processes. In particular, a number of researchers consider the key mission of communities as an influence on the internal development of professional communities and their members, including through the development of the qualifications system [1,2,3].
Another key area of activity is the spread of social and economic innovations, inclusive large-scale ones, which are possible both within the profession and outside it, in the general social space [4,5,6,7]. The involvement of experts in solving acute social and economic issues is not a new concept. However, in today's increasingly complex world, it is important to correctly cascade the analysis and discussion of ideas. That is why it is especially important to promote communities that are not only professional but also expert ones.
This study is a continuation of the authors' work [8,9], where the object of the study is those professional communities that consider the expert activity as the one of their main activities.
"How, wherefore and why do people unite their efforts and in which cases the experience of such associating is successful, and in which not?" [10] -the answer to this question interested researchers as far back as several centuries ago, initiating discussions about "the creation of 'the science and art of associating' at some time in the future".
The purpose of this study is to assess the performance of professional expert communities (hereinafter -PECs) based on key criteria and indicators which are representative for successful PECs.
Before moving on to the empirical part of the study, let us clarify the authors' understanding of successful communities, based, among other things, on the approaches of Russian and foreign researchers [11,12] to the formation of an assessment rating for the maturity of communities, to the perception of the community success through the prism of embedding institutionalized myths and elements of legitimization [13 ], to the assessment of community provision with a complex of resources as a criterion for their success [14].
When developing approaches to assessing the performance of PEC, we proceed from the fact that PECs are primarily characterized as a social rather than an economic phenomenon, therefore, when assessing PECs, the research toolkit mainly includes performance indicators understood as the achievement of socially significant results [15].
When constructing the toolkit for PEC assessing, all communities were divided into two types:  developing PECs that have existed for less than 3 years and are characterized by the absence of legitimization elements, not being equipped with material, technical or economic resources. Nevertheless, such communities may well be successful in achieving the outcome measures that are possible for a given stage of development and ensure meeting a set of criteria.  mature PECs, existing for more than 3 years, which are characterized by the provision of institutionalization elements and the necessary resources. The assessment of the success of such communities is carried out according to the maximum list of possible performance indicators from both economic and social points of view. During the study, hypotheses were formulated: 1. Successful PECs are characterized by the fulfillment of a set of criteria, among which economic and material-technical ones are manifested to the least extent in comparison with the block of social criteria (human, expert, communication).
2. Successful PECs with a high level of maturity are most active in the implementation of non-commercial expert activities aimed at solving socio-economic problems both at the regional and macro levels.

Methods of the Study
An empirical study of PECs activities includes the implementation of the following stages: Stage 1: search for PECs to be assessed. Since the object of our research is successful PECs, whose activities are aimed, among other things, at supporting the education system and the National Qualifications System (hereinafter -NQS), as a method of collecting information, we used the analysis of open sources containing data on the membership of Councils for Professional Competence (hereinafter -CPC). The selection of the surveyed CPCs included 25% of all Councils which are currently operating in the Russian Federation and representing different areas of professional activity: personnel management, finance, hospitality, information technology, office activities, engineering surveys, urban planning, architectural and construction design, beauty industry, agro-industrial complex, healthcare, electrical energy industry.
Stage 2: selection of PECs from the total number of communities represented in the  which correspond to at least four of the six criteria for PEC identification, developed by the authors in the framework of previous studies [8]: there is a professional group that positions itself as a professional community; there is a formal institution for the maintenance and development of community activities; the community positions itself as an expert one; the commercial activities of community members are also related to their expertise; there is a selection into the community and control of the professional competence of community members; community members are recognized experts.
The assessment of compliance with the criteria identifying the status of the PEC was carried out using the method of qualitative content analysis based on the analysis of the presence or absence of semantic parameters of the above criteria in the published open sources of the PEC (website, social networks).
Based on the results of the analysis, it was revealed that 50% of all studied communities meet the PEC criteria. In absolute terms, 56 such communities were found. Further, taking into account the above approach to the definition of mature and developing PECs, 18 communities were selected. 9 of them belong to mature PECs and 9 more -to developing ones.
Stage 3: an empirical study of the experience of successful professional expert communities based on the comparative case study method and interviews with community representatives.
The analysis of the structural characteristics of mature PECs showed that communities with an interregional level prevail (67%). The average period of their existence is 20 years, the number of community members varies greatly (from 10 people to 26,000 people), while the share of experts in such communities averages about 28%. The insufficiently high percentage of experts in the PECs is due, among other things, to serious requirements for the selection of experts. A representative of one of these PECs described the procedure for selecting experts as follows: "The position of an expert in our community is held for good reasons (education, public activity, implemented projects, etc.) and is constantly confirmed." Among the developing PECs, on the contrary, the main number of communities exists at the level of a separate region, the period of community activity is on average 1.8 years, the average number of PEC participants is 49 people, but at the same time, the average share of experts in the total number is higher than in mature communities (54.3%).
In order to use the comparative case study method, the authors have developed a toolkit for assessing the PECs activities whose application is possible for different empirical objects with different levels of maturity. The toolkit includes a scoring scale based on six criteria (Figure 1). It should be noted that the criteria reflecting the social and expertprofessional phenomenon of PEC are of the greatest importance in the assessment scale. The share of economic, material and technical criteria is in total no more than 20% for mature PECs and no more than 10% for developing PECs. Each of the presented criteria was decomposed into a set of indicators. Below, as an example, a fragment of the indicators for assessing the PECs by the criterion "communication" is presented. As you can see from the table, all indicators can be confirmed by data from open sources. During the assessment of developing PECs, such an indicator as "the presence of specialized media" is not taken into account because its implementation is more typical for mature PECs.
Thus, at the first stage, the communities were assessed based on open sources of information; at the second, the information received was supplemented and, if necessary, corrected based on the results of interviews with representatives of the PECs selected for the study. At the interview stage, special attention was paid to the assessment of indicators which were not obvious, available or sufficiently reliable in open sources.

Results and Discussion
The results of the PEC assessment using the comparative case study method are presented in the summary As can be seen from Table 2, according to the assessment results, mature PECs received on average about 252 points, which corresponds to a fairly high percentage of achieving a set of indicators (84%). The fulfillment of the following criteria is observed to the greatest extent: expert -91.39% (determined by such indicators as presence in the community of non-commercial, expert types of activity; mentioning of the PEC in the media; citing studies of PEC members; participation of PEC members in the work of public councils, CPCs, etc.) and communication -87.94% (see the figures in Table 2).
The leader of the rating among PECs related to the typology of a mature community scored 290 out of 300 possible points. This community is interregional, has existed for 28 years, it has more than 9000 members, including 127 experts, whose register is presented on the community website. The community has a long history that is part of the culture and reflects the core values of the PEC. The community has professional councils that monitor and evaluate the results of professional activities in the country. On the basis of the community, a Qualifications Assessment Center was also created. According to the PEC representative, it is the high level of provision with various elements of institutionalization, as well as the presence of a successful and recognized leader in the professional community that are the key criteria for the success of the community. This thesis is confirmed by scientists who believe that "organizations that build rationalized elements legitimized in the society into their formal structure increase their own legitimacy, resources and ability to survive" [13].
As for the developing PECs, the percentage of performance indicators was significantly lower than the planned maximum size, on average, such PECs received about 124 points. It means that the indicators lag behind the maximum possible by 38%. It should also be noted that the following criteria are most typical for developing PECs: human -72.54% (determined through such indicators as presence of a leader; presence of a formed community culture; qualitative characteristics of the team and community members; NPS of community members; involvement of community members in community activities, etc.) and a criterion closely related to it -expert (62.67%).
The leader of the rating among the developing PECs was the community which scored 158 out of 200 possible points: one of the few having an interregional level, despite its territorial location in the Urals Federal District. This community has a small number of participants (about 20 people), while 90% of the participants are experts. According to community representatives, it is the small size of the PEC, the high level of trust between the participants, as well as the lack of legitimization that are the reasons for the success of the community. This position is confirmed by the opinion of a number of foreign researchers who argue that the size of the organization complicates internal relations, and the presence of a hierarchy, a clear distribution of functions multiplies problems at the boundaries of the division of labor [16,17].

Conclusions
The results of the study fully confirmed the first hypothesis regarding developing communities. Their assessment showed that the fulfillment of economic and materialtechnical criteria in these PECs is manifested to the least extent.
As for mature PECs, the hypothesis was confirmed only in terms of the material and technical criterion (% of implementation on average no more than 61%). At the same time, according to the economic criterion, expressed in the fulfillment of such indicators as the presence of a complex of involved community financing instruments; community income per one active community member, etc., there is a fairly high level of implementation (more than 80%).
The second hypothesis, establishing a direct link between the success of PECs with a high level of maturity, expertise, and their activity in the implementation of noncommercial expert activities aimed at solving social and economic problems, was fully confirmed. As the analysis of the cases showed, the communities occupying leading positions in the rating (the first three places), according to the assessment, received the highest score (10 points out of 10 possible) for such indicators as: "presence of noncommercial, expert activities" and "presence of a register of community experts involved in solving socio-economic problems, consulting, expertise". At the same time, in the descriptive characteristics of these communities in all cases under study, the information about the active interaction of the PECs with the NQS institutes, etc. is given.
In conclusion, it should be noted that the proposed approach to the development of the PEC assessment toolkit needs further study and experimental confirmation. The authors plan to develop the presented approach in terms of specifying indicators and their maximum values. This will require a larger selection of communities, including its expansion to similar communities abroad, whose activities are related to the implementation of socially significant socio-economic tasks for the region as well as for the country.