By Julian Dierkes and Orkhon Gantogtokh
Rankings of schools and universities were initially conceived to bring accountability and transparency to education. In this, they are similar to all kinds of rankings that are applied to countries, including Mongolia.
Yet, university rankings have also been heavily scrutinized for reifying aspects of educational quality and assigning arbitrary worth to many aspects that seem meaningful only in particular contexts. A UBC colleague, Michelle Stack, has recently published an edited volume that examines many different aspects of university rankings.
Given our at least cursory interest in university rankings, we were very surprised to see a headline announcing National Univ of Mongolia (МУИС) to be a global leader in the criterion of publications published with international collaborators.
МУИС дэлхийн шилдэг 25 их сургуулийн нэг боллооhttps://t.co/VSacvE55jN pic.twitter.com/gA7U1UjU6V
— ӨДРИЙН СОНИН (@DNN_MN) October 1, 2021
We had never heard of u-multirank, so it seemed wise to investigate the creators of this index, their funding, and their methodology.
U-Multirank
At first glance, the ranking seemed immediately suspicious because the website is largely dysfunctional. It loads very slowly and is amateurish in design and search capacities. Some of the frustrations associated with the website turn out to be by design, as U-Multirank aspires to provide a different kind of ranking and to avoid the standard league tables.
According to the U-Multirank website, the project,
is a multidimensional, user-driven approach to international ranking of higher education institutions. It compares the performances of higher education institutions – in short: universities – in the five dimensions of university activity: (1) teaching and learning, (2) research, (3) knowledge transfer, (4) international orientation and (5) regional engagement.
And elsewhere,
U-Multirank takes a different approach to the existing global rankings of universities. Firstly, it is multi-dimensional and compares university performances in the different activities that they are engaged in. […]
Secondly, U-Multirank does not produce a combined, weighted score across these different areas of performance and then use these scores to produce a numbered league table of the world’s ‘top’ 100 universities.
Fair enough. Approaching rankings in a different way is a worthy ambition in principle.
The sense that the project is quite legitimate was reinforced by the consortium that organizes it. For example, it includes the German Centrum für Hochschulentwicklung which we had previously come across, in part because of their prize for German university managers, but also their German university ranking. Funding is provided by the equally respected Bertelsmann Foundation, but also by the EU via the Erasmus+ program.
So far so good, maybe the website is just terrible.
However, another aspect was more concerning in terms of understanding NUM’s ranking: NUM is the only Mongolian university included in this ranking!
NUM’s Ranking
This is how U-Multirank sees NUM:
In this variant of a spider graph, the longer the bar toward the outside, the stronger the performance.
The webpage dedicated to NUM provides further details on the elements that go into the “sunburst” graph.
The weakest performances come in categories like patents and programs taught in other languages. The strongest performance is the one highlighted by Udriin Sonin in their headline, namely “International Joint Publications”. Here, NUM has a score of 93.4%. What does that mean and how might this score have come about?
Unfortunately, the ? that appears on the webpage to look like a link to an explanation does not seem to provide such an explanation. The methodology page about indicators provides the following definition: “The percentage of the department’s research publications that list at least one affiliate author’s address abroad.” So, 93.4% of NUM’s research publications list at least one co-author who is based abroad. Where does this number come from then? Unfortunately, U-Multirank does not provide the raw data for its ranking to be able to pinpoint this exactly. But as far as the sources of data page explains, this percentage must be derived from bibliometric data obtained from web of science.
In the Web of Science database (fortunately, we have access through the UBC Library), there are 1,212 publications since 1995 that identify NUM as the affiliation of an author. Since we’re looking at this data anyway, here is some quick information to be gleaned.
Top five disciplines that these articles are classified as belonging to: environmental science (128 articles), ecology (90), applied mathematics (75), plant sciences (62), mathematics (61). The top five authors are: Bayartogtokh B (62), Chuluunbaatar O (56), Boldgiv B (49), Khuukhenkhuu G (49), Batkhuu J (39).
These numbers and listing of authors clearly show that some research at NUM is being recognized by peers around the world, the hallmark of a modern research university.
If we restrict the search to “articles” as document type (excluding such publication types as “proceedings”), the number of publications goes down to 989 and zoology replaces plant sciences in the top five disciplines. The top five authors remain the same, though the order is mixed up.
But what about the percentage of international co-authors? The top five co-authoring countries are Japan, Russia, USA, Germany, China.
When we exclude all articles that have a country other than Mongolia listed we end up with 126 of the 989 articles. By this search, roughly 12.7% of all articles are listed as having only a Mongolian affiliation. In other words, 87.3% of articles have a co-author not identified with Mongolia. If we use the wider definition of all publications (total of 1,212) we get 177 publications that are not affiliated with Mongolia = 14.6% meaning that 85.4% seem to share an affiliation with an international location. These numbers are not far off from the 93.4% reported in the U-Multirank index.
This still raises the question of why so many co-authors with international affiliations? As the comparison of NUM with other universities in the index shows, NUM is unusual in this regard, so it would be good to understand what this means, and how it might relate to NUM’s efforts to support and advance research among its faculty.
Web of Science does not allow for a search by citizenship of the co-author as that is (naturally) not collected as citation information. There is no obvious systematic way to see how many of the 85% of publications that are co-authored with international authors are co-authored with Mongolian researchers based abroad.
If we look at the 36 publications that show an affiliation with Canada many of them – though obviously not all – are natural science publications that have very long lists of authors where a single Mongolian name is included in this long list, but toward the very end of the listing of authors, suggesting that the role of this co-author may have been closer to a research assistant or perhaps participant in fieldwork rather than in the analysis. This is entirely impressionistic, however, and driven by the plethora of publications with NUM authors in fields that typically have many names listed as authors for publications. Perhaps the 93.4% percentage is based on a calculation that uses the total list of authors as the denominator rather than the number of articles.
Whatever the exact nature of the calculation that leads to this figure of 93.4% that is included in the U-Multirank, there is probably some more thinking to be done about what this figure means and what it says about the research strength of NUM. NUM obviously did not create this measure and it is unlikely that their research promotion efforts were built around it, so understanding the dynamic behind these frequent co-citations would help us understand the nature of the research enterprise at NUM and other research-based universities in Mongolia.
According to U-Multirank, NUM is the strongest in the research criterion among 10 criteria, but it conflicts with the score of their research performance shown in the comparison table. In the research performance criterion, NUM has been ranked below the average (300-350 range) out of 416 Asian universities. However, they are the strongest in the international joint publications, as shown in the table below.
Source: https://www.umultirank.org/
Despite the fact that the research performance of NUM is below the average (D score) among Asian universities, NUM publications are more numerous than Mongolian Academy of Science by approximately 14% in the last 4 years in the Web of Science (MAS-394; NUM-537). This raises another question of why Mongolia’s biggest research institution’s research outcome is lower than a single university’s performance while they receive the biggest share of research budget of the country.
It is good to note that in 2018, there was a dramatic 50% increase in the number of publications at NUM (average of 140 publications per year compared to 80 publications in 2017). Since then, it has kept the same performance. However, compared to a university in the relatively similar context, for example, State University of Yerevan, whose publication performance is ranked average, they have more than twice as many publications than NUM in the last 4 years.
Therefore, does being ranked in top 25 in the world in international joint publication really mean that NUM performs well in research, considering their overall research performance and the limited number of universities included in this ranking?
Mongolian Higher Education Context
The higher education sector is seen as a crucial means to help accelerate sustainable economic and social development in Mongolia. In Mongolia, the first modern university, National University of Mongolia was established in 1942. Mongolia has seen a remarkable expansion in its higher education since 1990 when Mongolia became a democratic country. Between 1991 and 2021, the number of higher education institutions (HEIs) grew from 14 to 88, and the gross enrolment ratio in higher education grew from 14.0% to 66.0% (MES, 2021 [PDF]). The total number of students rose from some 20,000 to about 150000 during the same period.
In 2021 a total of 147,293 students enrolled in Mongolia’s 88 HEIs. 52.2% of the total students studied at state-owned HEIs, 43.1% at private HEIs, and 4.7 % at religious HEIs. By the type of higher education institutions, 42.0 % of the total HEIs are universities, 54.6 % institutes, 3.4 % colleges and 6.3 % branches of foreign HEIs (MES, 2021 [PDF]). In 2021, entrants into Mongolia’s higher education institutions were as follows: 80.9% at bachelors, 16.8% at masters, 2.2% at doctoral and 0.1% at diplomas levels (MES, 2021 [PDF]).
The total number of employees in the Mongolian higher education sector is 12,000, and 59.7% (7143) of it is full-time academics (MES, 2021 [PDF]). 59.9% of full-time faculty members work at state-run HEIs whereas 40.1% at private HEIs.
Like many other Asian countries, Mongolian HE responded in various ways to the global trends of higher education, ‘such as growing social demand, privatization, accountability, marketization, economic growth, and internationalisation’ (Hou, 2015, p.311). However, these major changes in the number of students, institutions, specializations, and degrees are having a great impact on the quality of higher education. It is quite paradoxical that Mongolia has one of the highest numbers of universities per capita and a higher level of university enrolment yet the country is falling behind in its research publications and skilled employees in the Global Competitiveness Index (WEF, 2019).
About Orkhon G
Orkhon has recently commenced her studies for a PhD in Education Studies at the University of British Columbia. She has been actively engaged with the higher education reform processes of Mongolia with her civic engagement, research activities, and involvement in national-level projects. She has led the higher education sub-committee of the Education Reform Movement, an NGO established in 2019 to address the low quality of education in Mongolia. She completed an MSc in Higher Education at the University of Oxford in 2016. Her professional experience includes positions at Higher Education Reform Project as a HE Specialist, the London School of Economics Enterprise as a Researcher, Mongolian National Council for Education Accreditation as a Research and Partnership Manager, and NUM and MUST as a Higher Education Consultant and Mongolian Academy for Higher Education Development as Executive Director.
Follow
