Drop in 2016 Corruption Perception Index Score Not Very Meaningful

By Julian Dierkes

As readers of the blog know, I have developed an interest on Mongolia’s position on various global rankings over the years. I have written about indices in methodological terms as well as to try to understand Mongolia’s results. I have also become involved in formulating Mongolia’s scores for a number of indices myself.

Transparency International’s “Corruption Perception Index” (CPI) is one of the more important global indices. Not only does it have a long track record with a respected international non-governmental organization, but it is also an index that is used as a building block for many other indices, in part because corruption is difficult to measure, and there are not many alternatives to the CPI.

However, I’ve also been somewhat skeptical of the Mongolia scoring in the past and have written some notes about previous CPI rankings in 2012:

• methods, comparisons, Mongolia score
• more comparisons

Mongolia in the 2016 CPI

Let’s look at what actually happened in Mongolia in the newest CPI that was released in January 2017:

Before I get into methodology and an attempt to explain the change in score, it’s worth noting that the movement in rank (-15) is fairly substantial, while the score hasn’t changed much. The obvious interpretation of this year’s CPI is thus that not much has changed in Mongolia, but a number of other countries are perceived to be less corrupt.

Methodology and Meaning

Note that the CPI is fundamentally an aggregate score, i.e. it takes data from other rankings (that is interpreted to be related to corruption), standardizes that data on a 0-100 scale, adds it all together, and, presto!, a CPI score is born.

Transparency International does provide the xls sheets of data used in the calculation so that we have a look at the data included for Mongolia. Since the score changed from 2015 to 2016, let’s look at what was included with what score in those two years.

WB CPIA = World Bank Country Policy and Institutional Assessment
WEF EOS = World Economic Forum Executive Opinion Survey
Global Insight Country Risk = IHS Global Insight Country Risk
BTI = Bertelsmann Foundation Transformation Index
IMD World Competitiveness = IMD World Competitiveness Yearbook
Rule of Law = World Justice Project Rule of Law Index
PRS Intl Country Risk = PRS International Country Risk Guide
EIU Country Rating = Economist Intelligence Unit Country
Freedom House Nations in Transit

In comparing the two years, let’s first note the additional data point for 2016, i.e. the Freedom House Nations in Transit score. Then let’s also note that there are several, relatively small changes in a number of the other scores. Scores that went down: WEF EOS (-4), BTI (-1), IMD (-2), EIU (-4). Scores that went up: Global Insight (+5), Rule of Law (+3), PRS (+1). If these are added together we get a -2 overall movement. This is then reflected in a shift in the aggregate average of these scores of 38 for 2016 and of 38.5 for 2015 (rounded to 39). This makes it very clear that the score shift from 39 to 38 is NOT particularly meaningful. If any of the component scores had been just one lower in 2015, the score would have been 38 already.

Face Validity: What Happened in Mongolia in 2016?

With the conclusion that the change in the score is very slight only, there really isn’t that much of a point to a closer investigation of face validity in light of events in 2016.

Conclusion

What global indices can do is provide a benchmarking against other countries and a rough tracing of longer-term trends.

The 2016 CPI is another example of a year-on-year change in score that will likely be reported (I tweeted about it myself), but that should probably be ignored as it does not seem to be substantively meaningful.

Follow