We continue with the second part of the interview with Yong Zhao that began on May 18.

(Archivoz) What do you think of the research evaluation reform in China? What is the influence to Chinese Universities and academics?

(Yong Zhao) The reform of scientific research evaluation closely relates to the construction of scientific research integrity system. It is the baton of scientific research development, allowing scientists to do their own work well and universities develop their education function and social service function.

In September 2020, general secretary of the CPC Central Committee Xi Jingping mentioned “Four Look Forward” (English) at the Scientists’ Form: “I hope numerous scientists and technology workers will take up this historical responsibility, determinedly look toward the frontiers of global S&T, look toward the main economic battlegrounds, look toward major national needs, look toward the people’s lives and health, and incessantly march forward in the breadth and depth of S&T”. These four aspects are most important. The value of your scientific research, regardless of basic knowledge or application field, actually depends on whether it can solve the problems related to these four aspects.

A big difference between China’s scientific research evaluation and that of the western countries is that it relates directly to the salary of scientific researchers. Chinese universities have staff performance assessment every year. If you fail to meet the standard, your performance salary will be deducted. Unlike the contract system, the annual performance assessment system is critical relevant to the interests of scientific researchers, and it is highly concerned.

(Archivoz) About the evaluation of scientific research, are there any differences between China and other countries?

(YZ) The differences of research evaluation system between China and other countries essentially depend on the national policy, research environment, cultural differences and the economic development. It is not easy to make a comparison. The United States also experienced academic misconduct and fraud, which urged its system construction and formed a good research environment. The withdrawal of Haruko Obokata’ paper in Nature and the suicide of her tutor are driven by Japan’s scientific research environment and culture. Scientific research evaluation is an important part of the construction of national innovation system, and is closely integrated with R&D (Research & Development) level, research investment and the S&T strategy. We should not ignore these factors when we compare China and other countries.

China’s scientific research system is government-led national system, which is quite different from the foreign countries. The R&D investments between China and other countries are also different. After all, China is still a developing country, and its annual investment in R&D is not as much as that of western developed countries. China need to put the money where it is needed most. Therefore, the assessment of scientific research evaluation will be very strict. However, I do not think the evaluation of scientific research in other countries is more lax than that of China. When I was a doctoral student at Wageningen University & Research in the Netherlands, I learned that the evaluation indicators of the tutors in this university include the time of guiding students, the amount of papers published by tutors and students, and so on. As a policy and tool, scientific research evaluation is used all over the world. We are wrong to assume that the countries outside China are relaxing and free, some countries are even stricter than China.

In a word, we should look at the differences from a more macro perspective. Simple comparison is meaningless. Moreover, the reform of China’s evaluation system is in line with international standards, which is a major trend. For example, the time of evaluation cycle of UK REF and Australia ERA is relatively long, and China is extending the cycle time too.

(Archivoz) How do you see the evaluation of scientific research going in the near future? What are the opportunities looking ahead for the evaluation of scientific research in China Higher Education?

(YZ) The future development of China’s scientific evaluation relates closely to the “Two Centenary Goals”. Scientific research evaluation is heteronomy, and the evaluation came from the government and the University. While self-discipline means that, researchers will devote themselves to work and do not need to be evaluated externally. The balance between them depends on the country’s strong material base and GDP level.

When we achieved “Two Centenary Goals”, I believe the scientific research evaluation at that time is totally different from today. With the enhancement of scientific research strength and scientific research self-confidence, the role of scientific research evaluation (as heteronomy) in the development of science will become less important, whereas the self-discipline of researchers will play an increasingly important role. The intensity and complexity of many evaluation methods will be reduced. Researchers engage in scientific research by their interest and nature, which would be the perfect state.

In the new stage of development, China is facing many opportunities and challenges. With the development of S&T, such as aerospace, COVID-19, large equipment, etc., China gradually has a stronger voice in the world. In particular, with the global outbreak of the pandemic, China quickly developed the vaccine, which represents our S&T strength. How to protect “the sparks of fire”? How to ensure the healthy growth of scientific research? It is necessary to punish the misconduct of scientific research to ensure the fairness and justice of scientific research, which is exactly the role of scientific research evaluation. As a result, the reform policies on scientific research evaluation and integrity issued by the Ministry of Science and Technology, the State Council and the General Office of the CPC Central Committee are all new measures in response to the new era to ensure the long-term sustainable development of scientific research.

Furthermore, there is a “Punctuated Equilibrium Theory” in the field of public policy, that is, the policy change will be in the stage of stagnation and gradual change for a long time, but there will be a large-scale mutation in the short term, and the monopoly will be broken due to the change of public attention. In short, major focus events can lead to policy changes. At present, there are many major events in China’s S&T community, including positive events (enhanced S&T strength) and negative events (academic misconduct). From the perspective of policy and system construction, these events are great opportunities for the reform of scientific research evaluation system.

(Archivoz) Do you have any other thoughts or suggestions?

(YZ)

1. From an international point of view, the top four rankings have done very well. From the domestic point of view, how to improve the national scientific research evaluation system? How to improve the evaluation system of each university and department further? This is a hierarchical issue. We cannot simply focus on the macro or middle level. I think more problems are at the micro level, so scientific evaluation system must be stratified. Currently China does not consider stratification in scientific research evaluation, but focuses more on classification, such as the classification between comprehensive university and professional university mentioned previously.
2. In this great era of change, China is playing a leading role in many fields and it is inevitable for China to expose some problems that other countries do not have. We cannot learn from the experience of other countries. Unlike US, its economists and sociologists promote the development of the whole society by studying economic system, social thoughts, etc. China’s current situation is problem oriented. A host of problems forces us to change, reform, formulate policies and find solutions. Government need to formulate sound policies, and academics need to explore new methods and theories.
3. Inter-discipline is an important form of scientific research and an important driving force of S&T innovation. How to evaluate inter-discipline? This is a very big challenge. In 2020, Ministry of Education of the PRC added the category “inter-discipline” to the list of national disciplines accessible for academic degrees. Inter-discipline became the 14th first class discipline. It is under construction and everyone is talking about what subjects it will cover. Since it has become a discipline, we have to evaluate it. However, how to evaluate it? This is definitely a great challenge. Many attempts have also been made in the field of Library and Information Science. In recent years, the National Social Science Fund of China has funded many related projects, most of which remain at the theoretical level and lack practical application.

Interview conducted by: Ke Wu

We are interviewing Professor Yong Zhao, the director of Information Research Center of China Agricultural University, a Research Librarian and doctoral supervisor at the CAU Library. He has a PhD degree in Management and he engaged in postdoctoral research in Information Science in Institute of Scientific and Technical Information of China (ISTIC), the research direction of which was “Evaluation of Scientific Research”. Moreover, he is the deputy director of Development and Planning Office in CAU, managing the subject evaluation reform, the talent evaluation reform, etc.

He mentioned that evaluation of scientific research is a very important area of Library and Information Science, which is not only the evaluation of scientific research, but also the evaluation of all aspects of science, including the evaluation of science & technology, the evaluation of people, the evaluation of institutions, etc. From theoretical research to practical application, he has been practicing the policies and documents from the government and the Central Committee of the Communist Party of China (CCCPC).

(Archivoz) What is the current situation of the evaluation system of scientific research in China Higher Education?

(Yong Zhao) In the current situation of higher education evaluation system in China, 2016-2017 is a very important period for China’s policy reform. Why? There are two big backgrounds.

2016 is an important year, when China issued a series of documents on talent evaluation reform, including one document ‘Opinions on Deepening the Reform of the System and Mechanism for Talent Development’. This document first proposed “Overcome the orientations of ‘Academic Credentials Only’, ‘Professional Titles Only’, ‘Papers Only’, etc., and paper is not regarded as the restrictive condition to evaluate the application-oriented talents”. These documents were against “SCI Supremacy”, and caused strong repercussions in Chinese academic community.

In 2017, Springer announced the retraction of 107 papers published in Tumor Biology. Most of these papers were from China. In fact, there were some retractions in 2016, which were not as many as 2017, and had not caused enough attention in the society at that time. Why did this happen? Because “SCI Supremacy” has distorted the whole direction of scientific research. Therefore, the reform of scientific research evaluation closely relates to the construction of scientific research integrity system. This is another important background in China.

Policies and documents of CPC Central Committee are instructive and have the legal effect. It is the universities’ responsibility to implement these polices. In October 2020, the CPC Central Committee and the State Council of PRC issued “The Master Plan to Deepen the Education Evaluation Reform in the New Era” (English), which is currently the most authoritative document. The master plan put forward the reform goals for China’s education field, including improving evaluation of higher institutions (universities and colleges). Inheriting the previous policies, it proposed that “effectively address deep-rooted problem ‘Five Only’ (Scores Only, Enrollments Only, Academic Credentials Only, Papers Only, Titles Only)”, “promote the classified evaluation of higher institutions, guide the scientific orientation of different types of universities, and establish their own characteristics and levels”, etc. This is current situation and strategic orientation.

The so-called “classified evaluation” is to adopt different evaluation system according to different disciplines and industries. The medical evaluation reform is the first and fastest in the whole process, due to the high frequency occurrence of retractions and academic misconducts in this field (especially oncology and tumor biology).

The reform of university evaluation system also relates directly to the welfare of researchers. For example, China Agricultural University, for the first time in history, did not conduct professional title evaluation by the end of 2020, but in March 2021. The reason was that the relevant documents continued to be put off (a delay of 3 month) due to heated discussion. First of all, the professional title evaluation should break with “Five Only”, including “Papers Only”, “Titles Only”, “Academic Credentials Only”, “Prizes Only”, and “Projects Only”. The title evaluation system constructed by CAU reflects the diversification, instead of focusing on one indicator. Paper is just one aspect, and not a prerequisite or a mandatory condition, which is a significant change. This change brings new challenges and problems to the researchers, and determines the focus of their future work. On February 26, CAU issued a document on the evaluation of professional titles, and everyone is talking about the details of it.

In conclusion, scientific research evaluation is a major event in China’s higher education, which relates to the personal benefit and vital interest of every scientific researcher. It is the responsibility of scientific research management to make the achievements of scientific researchers be fairly recognized.

(Archivoz) What are the main problems of the current evaluation system?

(YZ) One of the most important problems of the previous evaluation system is “Papers Only”, which is not the main problem anymore. What problems will we encounter after the reconstruction of the scientific research evaluation system?

1. One aspect is about global common issues.

The scientific research evaluation system is newly constructed and needs time to adapt. There definitely will be some problems, especially in some evaluation methods. For instance, the evaluation of social services. There are mature bibliometric indicators for the evaluation of general scientific research, such as papers, patents, funds, etc. However, how to measure the scientific research contribution of rural revitalization, social services, regional development, and national strategic support? This kind of evaluation and methodology about social dimension and measurement are challenging, not only in China, but also in the whole world. From the perspective of Library and Information Science, it is indeed an invisible global challenge to evaluate social contribution.

The university has three functions: education function, scientific research function and social service function. For a long time, people equate the function of scientific research with everything. This is partly cause by some university rankings, like the top four rankings: US News, THE, QS, and ARWU. Besides scientific research, the evaluation criteria of these ranking lists also take into account the teaching, but does not include any indicator about social service. There is no weight for social service indicator.

How to measure the social service function? This is a very important methodological issue. There is no quantitative indicators in the current evaluation system. In the mature scientific research evaluation frameworks such as UK REF and Australia ERA, social service evaluation relies on cases and expert evaluation. These qualitative evaluations lack quantitative evaluation methods, and are easily questioned by participants.

2. The other aspect is about our Chinese situation.

We often directly apply the international ranking system, like the top four rankings, and simply equate the international excellence of the universities with their contribution to our country, without considering the national conditions. This is logically problematic.

The indicators of the international ranking lists are relatively rough, because they need to ensure an international lateral comparison. Thus, these indicators are internationalized, unified and standardized, and cannot be refined, differentiated and personalized. Some localized indicators and regional unique indicators cannot be included in the international scientific research evaluation system. However, it is these personalized indicators that actually play a crucial role in measuring the contribution of the universities.

The reality is that the local evaluation system, although combined with national conditions, such as Golden Apple Ranking, is immature and not highly recognized by the society. Most university presidents still focus on the international rankings. Therefore, many Chinese universities adopt the international evaluation indicators to implement the “Double First-Class Plan (World-class Universities and First-class Disciplines)”, which is one of the main problems of the scientific research evaluation in China.

In the evaluation system, we also lack certain scientific methods and data.

To build localized scientific a research evaluation system in China, we need the guidance from authoritative institutions, as well as fair, just and open data sources. Here is an example. The Ministry of Education has a statistical yearbook, which published the annual data of transformation of scientific and technological achievements, projects, etc. of all Chinese universities. After 2017, the data dimension of the yearbook has been changed from “University” to “Region and Category”. Now we can only find the transformation of S&T achievements in one region or one category, instead of one university. The Ministry of Education should have the universities’ data, but it did not consider the application value of these data in scientific research evaluation. It published the data from their own perspective, which invisibly expanded the data dimension and had a certain impact on our statistical analysis.

In terms of method, there is no distinction between comprehensive university and professional university in scientific research evaluation. People realize that this is a problem but have not found a solution yet. You cannot compare Peking University (comprehensive university) with China Agricultural University (professional university), which is unreasonable. The two universities have different goals and different disciplines. Let me make another example: China Agricultural University and Beijing University of Posts and Telecommunications. CAU mainly serves agricultural industry, while BUPT mainly serves information technology industry. Compare the transformation of S&T achievements of the two universities, CAU is not as good as BUPT. Nonetheless, CAU also bear the responsibility for rural revitalization and the rural modernization in China. Many teachers are engaged in public welfare and social services, such as Professor Xiaoyun Li’s poverty alleviation project in riverside village. It is easy to ignore the social service function of university, which is an urgent problem to be solved.

In addition, the research evaluation system is more about effectiveness evaluation, and lacks of performance evaluation. It is not good to only focus on the output. We should comprehensively consider the input and the output. For example, the input of a SCI paper in CAU and the input in Peking University are quite different. The results of scientific research evaluation directly relate to the national financial allocation of colleges and universities, so it is very necessary to distinguish the evaluation object and adopt the performance evaluation method.

Interview conducted by: Ke Wu