Option III. Recidivism Risk Algorithms
Recidivism risk algorithms are systems that generate a risk score for judges to reference when sentencing a defendant, predicting the likelihood of reoffending in order to increase the objectivity of sentencing decisions. However, Cathy O’Neil cites this algorithm as one of the cases of Weapons of Math Destruction. O’Neil points out that the data these algorithms use and learn from already carries embedded biases toward certain races and social classes (Talks at Google, 2016). Furthermore, because the algorithm’s formula is not disclosed, defendants have no way of knowing on what basis they were scored, and judges likewise do not clearly state how much weight they place on these scores. As a result, she explains, particular races and classes end up being unfairly classified as high-risk, and this classification generates a destructive feedback loop.
Although I am not a school teacher, these recidivism risk algorithms reminded me of my own school years. This may be somewhat difficult to imagine today, but at the start of each semester, under the pretext of understanding students’ family circumstances, we were often asked questions such as our parents’ occupations, which could roughly indicate the family’s financial situation, or whether we lived with both parents. In addition, there was a group of children labeled as “problem children”. Teachers paid closer attention to them, scolded them more often, casually called them “problem kids,” and when these children did act out, reacted as though this only confirmed what they had already expected. There was no formal risk-scoring system at the time, but if this information had been entered as values into a formal risk-scoring system, it would have led to intensified surveillance and discipline toward the students flagged as high-risk. While reading the materials for IP #3, these classmates came to mind. It is possible that they genuinely engaged in problematic behavior more often, but it is also possible that teachers simply paid closer attention to whether they misbehaved and scolded them more frequently as a result. Could it be that the label of “problem child” itself produced more of the very problems it was meant to describe?
What my past experience and recidivism risk algorithms share, yet also differ in, is that automated algorithms are dressed up in the impressive language of mathematical and statistical calculation. Because decision-makers are simply following the algorithm’s pre-calculated probabilities, they are more likely to fall into the false belief that their decisions are correct and objective, which can obscure or even deepen existing social inequality. When such tools are used to make important decisions, especially decisions that can change the course of someone’s life, there is a real risk that they become a convenient excuse for decisions that automate and accelerate social inequality, without that risk itself being adequately considered.
It is common to hear the argument that big data merely reflects the real world, and that learning from biased data is therefore unavoidable. While big data does reflect existing human biases and historical data, context is stripped away in the process of training on that data, and it is treated as nothing more than a simple data point (Noble, 2018). In that process, algorithms trained on biased data readily generate similarly biased and unfair data at scale, and, built on that foundation, further distort and amplify reality. Therefore, it should be noted that even if big data reflects reality, the algorithms built through it cannot be described as neutral.
Algorithms cannot make new choices beyond the data they are given. We, as human beings, however, can make new choices and correct what is wrong, even when past cases are biased and even when the real world itself is biased. I believe this is the essence of human agency. Education, too, is not simply about learning the past as it is, but education is about creating new meaning from a variety of sources. The convenience and benefits that big data offers are undeniable, and we should make use of the many opportunities it can bring. But while big data and algorithms excel at accelerating the process of decision-making, what education needs is not efficiency and speed but fairness, ethical responsibility, and the kind of contextual depth that comes from working through a complex and difficult process to arrive at one’s own answers. For intelligent and ethical uses of big data, I believe it is important that we do not treat tools like big data as neutral, but instead understand and acknowledge their limitations, and maintain a sharp, critical perspective so that “human agency” is never excluded, whether in how we use these tools or in the decisions we make with them.
References
Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. New York University Press.
O’Neil, C. (2017, July 16). How can we stop algorithms telling lies? The Observer. https://www.theguardian.com/technology/2017/jul/16/how-can-we-stop-algorithms-telling-lies
Talks at Google. (2016, November 2). Weapons of math destruction | Cathy O’Neil | Talks at Google [Video]. YouTube. https://youtu.be/TQHs8SA1qpk
