Evaluate the impacts of researchers in China with a novel indicator framework

. Evaluating the impacts of researchers plays a role in identifying impactful researchers, cultivating talents, and promoting talent exchange. Traditional indicators emphasize researchers’ scholarly impacts and rely on bibliometric data, which take a long time to reveal the impacts. With the popularization of social networks, researchers have gone beyond academia and shown their impacts instantly on the general population. Although altmetrics have been proposed to measure the societal impacts of researchers, they show differences across countries and regions. A comprehensive indicator framework for evaluating the impacts of Chinese researchers is lacking. This study proposes a novel indicator framework based on bibliometrics and altmetrics and uses it to evaluate the impacts of researchers in China. Specifically, the proposed framework consists of 2, 3, and 17 first-level, second-level, and third-level indicators, respectively. We conduct a case study with data from various online platforms. Results demonstrate that the indicator framework can evaluate the scholarly and societal impacts of Chinese researchers. The results also show that researchers’ societal impacts are stronger than their scholarly impacts in China. According to the impacts, the indicator framework can categorize researchers into different groups, among which the largest group contains ordinary researchers with mediocre scholarly and societal impacts.


Introduction
Cultivating impactful researchers plays a role in the era of global competition. However, the impacts of researchers are not directly observable as research input and output are not linearly related and researchers may impact others in different ways. With the rapid development of disciplines and research areas, building an indicator framework for evaluating the impacts of researchers has become a critical issue in research management. An effective indicator framework enables the evaluation of researchers' impacts and subsequent research management activities, such as assessing researchers and recruiting talents.
Given the importance of evaluating the impacts of researchers, many studies have developed evaluation frameworks based on bibliometric data, such as the number of publications, citation counts, and h-index [1,2]. Bibliometric indicators generalize to different countries and regions, but they are limited to evaluating scholarly impacts in a delayed manner. With the development of information communication technologies, research outputs can be diffused promptly and widely to the whole society [3]. To measure researchers' societal impacts instantly, more recent studies have proposed altmetrics, which are indicators based on various user activities in social media environments [4,5,6,7]. However, the effectiveness of altmetrics differs across countries and regions [8,9,10]. Besides, few altmetric indicators generalize to different counties and regions [11,12,13]. This is especially true for China because many mainstream social platforms (Research Gate, Facebook, Twitter, etc.) are barely accessible in China. Besides, the Chinese prefer indicator frameworks rather than individual indicators or evaluation methods when evaluating the impact of researchers. However, studies on constructing indicator frameworks for evaluating the impact are scarce. Consequently, there is a need for an indicator framework that can comprehensively evaluate the impacts of researchers in China.

Literature review
The impact of researchers refers to the breadth and depth reached by the researchers' academic activities and research outputs during a period. The impact of researchers stems from their research outputs' quantity and quality, which are produced through formal academic communications [14,15]. The main indicators include the number of papers published in the core collection, the number of papers published in ordinary journals, the number of cited papers, the impact factors of journals, etc. With the development of academic communication systems and the wide application of social media, some researchers have paid attention to the impact diffused by informal academic communications and emphasized the necessity of evaluating both the scholarly and societal impacts of researchers [16,17].
Early studies used peer review, citation analysis, and other bibliometrics-based methods to evaluate the impact of researchers. However, previous methods have several problems, including subjective evaluation indicators, insufficient data sources, inefficient evaluation processes, and incomplete types of research outputs. To overcome the problems, researchers have turned attention to altmetrics, which are new online indicators developed in the context of online academic communications. Regarding the construction of indicator frameworks for evaluating researchers' impact, previous studies mainly used the methods of the hierarchical index, principal component analysis, co-occurrence matrix, and correlation analysis to evaluate the impact.
With the emergence of Web2.0, social media platforms enable recoding researching behavior and academic tracks. In such a context, the concept of altmetrics has been developed and attracted much attention from academia. Altmetrics reflects the scholarly and societal impacts of research outputs in a timely manner by collecting online indicator data, such as links, articles, and blogs [4,5,18]. Some researchers believe that indicators and indicator frameworks based on altmetrics are new ways for evaluating the impact of researchers.
In the last decade, studies on altmetrics focus on two aspects: correlation supplement analysis between altmetric and traditional indicators and the application of altmetric indicators in evaluating researchers' impact. Cheverie et al. [19] proposed that online evaluation has been an important tool for academic evaluation and online data (social networks, open data platforms, bookmarks, and publishing rewards) can be used to evaluate researchers' impact. Priem and Hemminger believed that altmetrics plays an important role in bibliometric works and provides new indicators for evaluating the value and impact of scholarly works [4]. Altmetrics also addresses the incompatibility and limitations of traditional bibliometric indicators in the Internet era. Taraborelli [20] stated that peer review fails to meet the authority requirements in many research areas and believed that it should be supplemented by soft peer review because they are correlated.

Methodology
This study involves qualitative analysis and quantitative analysis. On the one hand, this study uses a literature survey to collect relevant studies from various academic databases and online platforms and obtains guidelines and references from the studies. We also interview researchers to obtain their opinions about indicators of researchers' impacts. During the process, we refer to research results at home and abroad and the opinions of different researchers, select suitable indicators, and build our indicator framework for evaluating the impacts of researchers in China.
On the other hand, we use multiple quantitative analysis methods to quantify the impacts of researchers in China, including entropy weight, correlation analysis, and matrix analysis methods. When obtaining the weights of indicators with the entropy weight method, we standardize the indicators' value, calculate the relative value and information entropy of the indicators, and calculate the weights based on the relative value and information entropy. When conducting correlation analysis, we use the Pearson correlation analysis method to analyze the correlation between the number of research outputs and the impact of research outputs and the correlation between the number of research outputs and the diffusion of research outputs in social networks. When conducting matrix analysis, we use the scholarly and societal impacts as two analysis dimensions and categorize researchers into four groups according to their scholarly and societal impacts. Then, we analyze the indicators' distributions in each researcher group. The following presents the details of the quantitative analysis.
The entropy weight method The entropy weight method is based on the information theory and measures the weights of indicators according to their dispersion degrees. The greater the dispersion degree of an indicator, the greater the influence of the indicator on the framework's overall outcome, and the greater the weight of the indicator. Subjective weighting methods face the problems of randomness, assumption, and information overlap among multiple indicators; the entropy weight method can address the problem effectively. For each indicator, the entropy weight method calculates its weight as follows: where is the value of the j-th indicator of the i-th researcher, is the normalized value of , m is the number of levels in the indicator framework, , , and are the information entropy, information redundancy, and weight of the -th indicator, respectively. Panel data matrix with multiple indicators Following previous research [21], we use matrix X to annotate panel data with multiple indicators and normalize its elements to eliminate the difference of scales between indicators: where ij X indicates the value of the -th indicator of the -th researcher, ij ' X is the normalized value of ij X , and ij max X and ij min X refer to the maximum and minimum ij X respectively.

Data processing
The main content of this study includes selecting appropriate measurable variables, constructing the indicator framework for evaluating the impacts of researchers in China, and illustrating the effectiveness of the indicator framework with relevant data.
To showcase the proposed indicator framework, we select Chinese researchers from the fintech field, a burgeoning and influential research field in China. We use the HowNet database, WanFang database, ScienceNet, WeChat public accounts, Baidu encyclopedia, and other online platforms to collect data related to the selected researchers. We use Excel, SPSS, and other software to process the raw data and conduct statistical analysis. After that, we analyze the correlations between second-level indicators. We also conduct matrix analysis and categorize researchers into different groups to provide guidance for enhancing their impacts.

Data descriptions and statistics
We selected 98 researchers with senior professional titles in the Fintech domain. They have successfully applied for projects from the National Natural Science Foundation of China or the National Social Science Foundation of China between 2014 and 2019. These researchers constitute a valid sample set as they represent the group of Fintech researchers with high impacts [22]. After removing duplicate researchers or researchers with missing data, we obtained 77 valid samples. The data were collected between 14th May and 17th May 2021. After collecting the data, we analyze the coverage of each indicator. We found that, among all the indicators, the coverage of the number of national projects, The number of cited papers, the number of citations, the average number of citations per paper, H-index, the number of mentions in WeChat public accounts, and the number of downloads in CNKI has reached 100%. This means that the Chinese Fintech researchers have good coverage of these indicators. However, they have low coverage on the number of recommendations in ScienceNet and the number of comments in ScienceNet. The coverage is 10% and 9%, respectively. The results show that the researchers need to improve their activity of the two indicators.

Weights of indicators
We calculate the weights of indicators using the entropy weight method and present the weights in Table 1.
We also conduct correlation analysis between the goal and each of the second-level indicators. The results show that the impact of researchers in China is positively correlated with the number of research outputs, the impact of research outputs, and the diffusion of research outputs in social networks. The correlation between the impact of researchers and the diffusion of research outputs in social networks is 0.768, which is the strongest. This means that the diffusion of research outputs in social networks influences the impact of researchers the most among all the second-level indicators. The impact of research outputs comes next as its correlation with the impact of researchers is 0.685. The correlation between the number of research outputs and the impact of researchers is 0.506, meaning that they are also positively correlated. The correlations between the goal and each of the second-level indicators are significant at the level of 0.01.
Among the second-level indicators, the number of research outputs and the impact of research outputs have a significant and positive correlation (r = 0.562), while the diffusion of research outputs in social networks has positive but insignificant correlations with the others. The diffusion of research outputs in social networks measures the societal impact and the other two second-level indicators measure the scholarly impact. The results indicate that the two first-level indicators do measure different impacts of researchers.

Two-dimensional matrix analysis
Derived from Boston matrix analysis, two-dimensional matrix analysis constructs four quadrants based on two dimensions and positions subjects into corresponding quadrants [23]. Therefore, we use the two-dimensional evaluation method to further analyze the impacts of Chinese Fintech researchers. Following previous research [24,25], we consider the scholarly and societal impacts as two analysis dimensions and categorize researchers into four groups according to their scholarly and societal impacts. The four groups are famous researchers, star researchers, professional researchers, and ordinary researchers.
Before conducting the two-dimensional matrix analysis, we need to transform data. Based on the principle of 20/80, this paper selects the top 20% of researchers with the highest scholarly impacts as a dividing line for the scholarly impact. Similarly, we select the top 20% of researchers with the highest societal impacts as a dividing line for the societal impact. The scores of the scholarly impact of the former researchers are higher than 20.9955 while the scores of the societal impact of the latter researchers are higher than 16.6061. We consider the scholarly impact as the x-axis and the societal impact as the y-axis. After that, we subtract the dividing lines from the scores of scholarly and societal impacts for each researcher. Figure 1 shows the results of the two-dimensional matrix analysis.
The two-dimensional matrix can position researchers according to their scholarly and societal impacts and shed light on improving their impacts.
Famous researchers Researchers in the first quadrant are famous researchers who have high scholarly and societal impacts. Therefore, famous researchers have good research achievements and high societal reputations and can be considered prestigious researchers in their fields. There are 18 famous researchers in the Fintech domain. The number accounts for 23.38% of the samples and thus follows the principle of 20/80. Take Dexu He as an example. He is the president of NATIONAL ACADEMY OF ECONOMIC STRATEGY, a professor and doctoral advisor of the Graduate School of Chinese Academy of Social Sciences. He is also an expert of the National Social Science Foundation of China, a vice president of the China Rural Finance Association, and an executive director of the China Finance Association. He has published more than 200 papers in domestic and international academic journals, has successfully applied for more than 90 national and provincial research projects, has published 12 books, and has been mentioned almost 400 times in WeChat. His h-index has reached 39 and his papers have been downloaded 44318 times in CNKI. Consequently, Dexu He is a prestigious researcher in China. Besides, Shaobo Liu from Jinan University, Qiguang Ma from Shanxi university of science and technology, and Guanghe Ran from Chongqing university also have high scholarly and societal impacts.

Star researchers
Researchers in the second quadrant are star researchers who have high societal impact and low scholarly impact. There are 14 star researchers in the Fintech domain. The number accounts for 18.18% of the samples. For example, Bohui Wen is a vice president and a doctoral advisor in the school of finance, Tianjin university of finance and economics. Her papers have been downloaded 20815 times in CNKI. She has been mentioned in one of the posts in ScienceNet 16264 times. Besides, she has been recommended for 1 time and discussed 4 times in ScienceNet. However, she has published only 32 papers and her h-index has just reached 8. Her scholarly impact is relatively low compared to her high societal impact. Similarly, Guohe He from Wuhan university and Shangmei Zhao from Beihang university are popular on ScienceNet, WeChat, and CNKI, but they have relatively low scholarly impacts.

Ordinary researchers
Researchers in the third quadrant are ordinary researchers who have mediocre scholarly and societal impacts. There are 32 ordinary researchers in the Fintech domain. The number accounts for 41.56% of the samples. This group is the largest among the four groups. Fintech is a new research direction rising in recent years. The ordinary researchers have low scholarly and societal impacts, meaning that they only conducted a few Fintech studies and online activities to make themselves visible on various platforms. However, ordinary researchers are close to the origin of coordinates, which indicates that there is great potential to improve their scholarly and societal impacts. Yueli Xu from Zhengjiang university of technology, Supu Wang from Xinjiang University of Finance & Economics, and Changluan Fu from Hangzhou normal university are examples of ordinary researchers. Although ordinary researchers have low scholarly and societal impacts, they may grow into professional, star, or prestigious researchers as they devote more time and effort to their research.

Professional researchers
Researchers in the fourth quadrant are professional researchers who have a high scholarly impact and low societal impact. There are 13 professional researchers in the Fintech domain. The number accounts for 16.88% of the samples. Take Shoudong He as an example. He is the president and a doctoral advisor of the business school, Jilin University. He is also a director of financial measurement and risk management division, qualitative economics research center. He has successfully applied for or participated in more than 100 national or provincial research projects. He has published more than 120 papers, which have been cited more than 3797 times. His h-index has reached 28. Consequently, he has a high scholarly impact. Compared to the scholarly impact, his societal impact is relatively low. He has been mentioned only about 100 times in WeChat public accounts and his papers have been downloaded only 4041 times in CNKI. Zhiwei Su from ocean university of China and Fengge Yao from the Harbin Institute of technology are two other examples of professional researchers.
In addition to the above four groups, there is an outlier in the second quadrant as shown in Figure 2. Its score of the societal impact is 349.1622, which is much higher than those of other star researchers. The outlier is Hongwei Zhang. He is the director of academic affairs, a professor, and a doctoral advisor of Sichuan university. He is also a vice president of the Chinese Society for Macroeconomic Education and the Director of the Center for Public Economy and Public Administration of Sichuan University. He has an extremely high societal impact as shown by the number of downloads in CNKI, the number of mentions in WeChat public accounts, the number of mentions in ScienceNet, the number of readings in ScienceNet, the number of recommendations in ScienceNet, and the number of comments in ScienceNet. The numbers are 46750, 262, 10, 42037, 15, and 18, respectively. Besides the reports about his financial research, there are cases about his social status as the director of the academic affairs of Sichuan university. Such reports may enhance his societal impact and thus explain why his societal impact is much higher than other star researchers.

Discussion
Indicators of impacts are important for evaluating researchers. Previous evaluation frameworks are limited to scholarly impacts and rely on bibliometric data, which take a long time to reveal the impacts. However, research outputs can be diffused promptly and widely to the whole society through social platforms. Although altmetric indicators have been proposed to measure researchers' societal impacts. The altmetric indicators cannot generalize well to different counties and regions. Therefore, there is a need to build a comprehensive indicator framework for evaluating the impacts of researchers in China. We have proposed a novel indicator framework for the purpose and conducted a case study to demonstrate the framework's effectiveness using data about Chinese Fintech researchers. We have several discussions based on the case study.
First, based on theories and methods of altmetrics, this paper introduces a novel indicator framework, which involves researchers' informal academic communication data. The proposed framework consists of 2, 3, and 17 first-level, second-level, and third-level indicators, respectively. In the framework, many indicators are specialized to the Chinese context. For example, ScienceNet and WeChat public accounts are added as new data sources. The proposed indicator framework extends the existing research on researcher evaluation and meets the needs of evaluating the impacts of researchers in China.
Second, the societal impact is more important than the scholarly impact for Chinese Fintech researchers. In the empirical study, the ratio of the scholarly impact to the societal impact calculated based on the entropy weight method is 0.3952:0.6048, which is about 4:6. The result means that the Chinese Fintech researchers emphasize more on information academic communications. This is consistent with previous research [26,27]. When the societal impact outweighs the social impact, traditional indicators fail to measure the true impacts of researchers. Therefore, it is necessary and important to incorporate altmetrics into researcher evaluation.
Third, the proposed indicator framework complements the existing evaluation frameworks. In the empirical study, we conduct correlation analysis between the impact of researchers and each of the second-level indicators. The results show that the impact of researchers is positively and significantly correlated with the number of research outputs, the impact of research outputs, and the diffusion of research outputs in social networks. Their correlations are 0.506, 0.685, and 0.768, respectively. Among the three second-level indicators, the diffusion of research outputs in social networks has the strongest correlation with the impact of researchers, meaning that the societal impact reflected by information academic communication data is important to evaluating researchers in China. The results are consistent with previous studies on evaluating Chinese researchers with altmetric indicators [27,13].
Fourth, the proposed indicator framework can categorize Chinese Fintech researchers into four groups. This study considers the scholarly and societal impacts as two analysis dimensions and conducts a two-dimensional matrix analysis based on the two dimensions. The researchers are categorized into famous researchers, star researchers, professional researchers, and ordinary researchers. Among the four groups, ordinary researchers are the largest, which accounts for 41.56 of the samples. This means that most Fintech researchers have mediocre scholarly and societal impacts in China, which reflects the characteristics of new emerging research areas. Besides, ordinary researchers are close to the origin of coordinates. The results mean that the impacts of Chinese Fintech researchers follow the principle of 20/80 and the results are consistent with previous research [13,23].

Conclusion
We have several conclusions based on the analysis in the last section. First, this paper leverages bibliometric and altmetric data to construct an indicator framework for evaluating the impacts of researchers in China. The proposed indicator framework is novel and extends existing research on researcher evaluation in China.
Second, compared to the scholarly impact, the societal impact contributes more to the impact of researchers in China. This finding reveals the internal relations between different indicators of evaluating researcher impacts. Besides, the finding demonstrates the effectiveness of using multi-dimensional indicators and provides theoretical support to improving researcher evaluation in China.
Third, Chinese Fintech researchers can be categorized into four groups based on their scholarly and societal impacts. The result shows that the distribution of prestigious researchers and the other researchers follow the principle of 20/80. The result also demonstrates that dividing evaluation indicators into the scholarly impact and the societal impact is reasonable.

Limitations and future work
This study has several limitations. First, we only have 77 valid samples and the samples are selected because they have senior professional titles and have successfully applied for research projects from the National Nature or Social Science Foundation of China. Future research can select more researchers to enrich the samples. Second, some popular social platforms in China (e.g., Sina Weibo) are not considered in the proposed indicator framework due to the unavailability of relevant data. Future research can enrich the framework with more contextualized indicators and collect relevant data to evaluate the new framework. Third, papers may have multiple authors with varied contributions, but this study attributes all contributions to the first author or corresponding author. Future research can define the contributions of different authors and thus refine the impact of researchers. Fourth, evaluation indicators may have different effects across disciplines, but the proposed indicator framework fails to consider such differences. Future research can design specialized indicator frameworks according to the characteristics of disciplines.