Month: April 2025

Upstream Bias in Educational Data: Lessons from the Golden Record Exercise

The rapid advancement of AI represents one of the most significant developments in writing and reading technology in recent history. Just as the printing press revolutionized access to written knowledge in the 15th century, AI technologies are fundamentally transforming how information is created, accessed, and interpreted in educational settings. The United Nations Human Rights Council (2024) notes that “Generative artificial intelligence systems are newer forms of artificial intelligence… designed to produce diverse outputs on the basis of extensive training data sets, neural networks, deep learning architecture and user prompts” (para. 11), enabling the creation of text, images, and other content that was previously only possible through human effort.

It’s tempting to look only at the upside of this new technology, and how it addresses needs for speed, scale and personalization, but this would be to overlook the challenges it poses to equity and representation. As noted by the UN Special Rapporteur, “technology is never neutral – it reflects the values and interests of those who influence its design and use, and is fundamentally shaped by the same structures of inequality that operate in society” (UN Human Rights Council, 2024, para. 7). My experience with the Golden Record visualization exercise provides a revealing case study of how modern writing/reading technology systems can inadvertently perpetuate patterns of exclusion, even with seemingly neutral parameters.

The Power Dynamics of Data Curation

An experience I had in this course enabled me to see how data collection, under the guise of neutrality, can actually reinforce existing patterns of exclusion. What seemed like a simple engagement with a survey and the resulting visualization revealed deeper concerns about who gets represented in data systems.

The seemingly neutral, objective process of collecting and categorizing data contains inherent power dynamics that determine which perspectives are validated and which are excluded. For the Golden Record assignment, I listened to the Podcast episode and picked 10 tracks that were on the Golden Record, but not 10 ‘musical tracks.’ Among my 10 picks were “United Nations Greetings / Whale Songs” and “Sounds of Earth – Voyager Golden Record,” which were listed on the podcast page (the instructions did tell us to listen to the podcast, which I did). But I did not go to the hyperlinked Youtube page that had the 27 tracks from which we were supposed to make our selections. Moreover, the names of the tracks on the podcast page were different from those on the Youtube page.

My 2 non-music tracks were not represented on the survey, and I wasn’t sure if I was picking the right songs because some of them had different names than on the Youtube page. I didn’t want to spend all that time cross-referencing the songs on the podcast page with the songs on the Youtube page. I had already made my choices and submitted my assignment, but when it came time to do the task, I could choose a maximum of 8 because 2 of my choices were excluded, and one got lost in the shuffle—I couldn’t identify it among the Youtube names.

But then again, was it my problem, or was it an inflexible system that couldn’t accommodate different interpretations? This small-scale experience mirrors a much larger problem in AI systems: what doesn’t get counted in the data simply doesn’t exist in the resulting model. Just as my legitimate non-musical selections disappeared from the visualization because they didn’t fit predetermined categories, entire perspectives and knowledge systems from marginalized communities are systematically excluded from AI training data. The survey couldn’t accommodate my non-standard inputs—similarly, AI systems trained on limited or biased datasets cannot represent knowledge they’ve never been exposed to. This isn’t just about personal inconvenience; it’s about who gets to contribute to our collective digital knowledge and whose perspectives are systematically erased.

My experience with the Golden Record visualization reflects what Reiss (2021) identifies as the power dynamics embedded in digital data systems. The survey functioned as a tracking system that collected specific inputs but excluded others – it literally couldn’t account for my non-musical selections, even though they were legitimate parts of the Golden Record. This illuminates how data collection systems make assumptions about what “counts” as valid input (musical tracks only in this case), enforcing a kind of standardization that flattens innovation and variation. As Reiss explains, such digital systems—including AI in education—evolved to address legitimate challenges like providing individualized feedback at scale, yet they simultaneously reinforce specific modes of knowledge at the expense of others. In my case, the system’s inflexibility to accommodate legitimate diversity in interpretation meant that my non-musical selections became what the assignment prompt describes as “data that is missing, assumed, or misinterpreted” (assignment instructions).

In the data visualization that the survey results produced, I was underrepresented because most people had 10 choices and I only had 7. This small instance illustrates what Baker and Hawn (2022) describe as “representational bias” in educational data systems, though they examine this phenomenon on a much more consequential scale. Their research into AI systems like automated essay scoring reveals how these technologies may systematically disadvantage certain student populations. Their analysis demonstrates the critical importance of addressing bias as these tools increasingly determine how students’ writing is evaluated and valued.

The UN Special Rapporteur’s report highlights how data collection bias operates: “If particular groups are over- or underrepresented in the training sets, including along racial and ethnic lines, algorithmic bias can result” (UN Human Rights Council, 2024, para. 13). This upstream exclusion represents a form of systemic discrimination that occurs before an algorithm ever makes its first prediction or assessment. This upstream perspective is particularly valuable because it redirects attention to earlier stages in the AI development process. While much discourse on AI bias focuses on downstream effects—the outputs and predictions an algorithm produces—the more fundamental issue is that certain possibilities are excluded from consideration before the algorithm ever begins its work. When the inputs themselves aren’t collected or recognized as valid, no amount of algorithmic fine-tuning can recover these missing perspectives. This form of invisibility is especially pernicious because those affected may never know they’ve been disadvantaged, as the system presents itself as comprehensive despite significant blind spots in its foundational data.

Educational Consequences of Data Collection Bias

According to TeachFlow (2023), AI tools are changing education by giving students personalized learning, helping with paperwork, and providing quick feedback on their work. While these tools offer new ways to teach reading and writing, there are important concerns about how these systems might contain bias in their data and programming.

In educational settings, the consequences of biased data collection manifest in several concerning ways. Predictive analytics tools deployed in schools “often rate racial minorities as less likely to succeed academically and in their careers, because of algorithm design and data choices” (UN Human Rights Council, 2024, para. 44). The report further notes that educators may subsequently “steer students from marginalized racial and ethnic groups away from educational and career choices that would maximize their potential and offer the best opportunities to break cycles of exclusion” (para. 44).

When schools adopt AI systems built on incomplete or unrepresentative data, I’m concerned they do more than just maintain the status quo of inequality. These systems can actually intensify biases and make disparities worse for students from underrepresented groups. A single assessment algorithm can affect thousands of students’ educational trajectories, with life-altering consequences for individuals from marginalized communities.

The Challenge of Algorithmic Opacity

Compounding the issue is what the UN Special Rapporteur terms the “‘black box’ problem” (UN Human Rights Council, 2024, para. 21) of AI systems. As algorithms incorporate new patterns from the data they process, “individuals relying on the algorithm may no longer be able to ‘look under the hood’ and pinpoint the criteria that the algorithm has used to produce certain outcomes” (para. 21). As the UN report emphasizes, this opacity creates situations where discrimination becomes “increasingly difficult to identify, prove and remedy” (UN Human Rights Council, 2024, para. 21), particularly when algorithmic systems leave minimal evidence of how decisions were reached.

For students and educators alike, this creates a troubling scenario where consequential decisions are determined by systems whose reasoning processes remain inscrutable. Perhaps the most concerning aspect is that students themselves may never know why they were classified a certain way.

We created AI writing systems to meet our need for efficiency and scale, but these inventions are now shaping literacy practices in ways that can amplify existing inequalities. My Golden Record experience offered just a glimpse of what happens when systems exclude certain inputs – imagine this same pattern playing out in AI writing tools evaluating millions of students’ work, potentially misrepresenting entire communities through skewed algorithmic judgments.

Solutions/Mitigation Strategies

Addressing bias in educational data systems requires deliberate intervention at several levels. Baker and Hawn (2022) advocate for “representation-focused validation approaches” that examine whether algorithmic systems produce equivalent outcomes across different groups. While much of the discourse on AI bias focuses on technical solutions, the human element remains crucial in mitigating these issues. According to Teachflow (2023), “providing professional development opportunities for educators is crucial to ensure they are equipped with the necessary skills and knowledge to effectively integrate AI tools into their teaching practices.” When educators develop algorithmic literacy – the ability to critically assess AI systems’ capabilities, limitations, and potential biases – they can design learning experiences that reveal these issues to students. Indeed, my Golden Record experience appears deliberately structured to illustrate data collection limitations – creating a pedagogical moment that exposed how systems might exclude valid inputs. This instructional approach mirrors what Teachflow advocates: training that extends beyond technical skills to include critical examination of how data collection methods may encode existing inequalities. By designing experiences that allow students to personally encounter upstream bias in data collection, educators can cultivate the critical awareness needed to identify and address these issues in consequential real-world AI applications.

Addressing these challenges requires fundamental reconsideration of how educational data is collected, processed, and deployed. The UN Special Rapporteur recommends that stakeholders “ensure meaningful and effective consultation with those from marginalized racial and ethnic groups, professionals from relevant societal domains and those with expertise in systemic racism in the design, development and deployment of artificial intelligence products” (UN Human Rights Council, 2024, para. 69).

Additionally, the report calls for “human rights due diligence assessments at all stages of artificial intelligence product design, development and deployment” (para. 69), emphasizing that upstream interventions are essential to preventing downstream harms. The core challenge involves questioning basic assumptions about data collection practices rather than merely addressing algorithmic outputs after biased data has already shaped the system.

References

Baker, R. S., & Hawn, A. (2022). Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32, 1052–1092. https://doi.org/10.1007/s40593-021-00285-9

Reiss, M. J. (2021). The use of AI in education: Practicalities and ethical considerations. London Review of Education, 19(1), 1–14. https://doi.org/10.14324/LRE.19.1.05

Teachflow. (2023, June 30). Assessing the reliability and bias of AI in education. https://teachflow.ai/assessing-the-reliability-and-bias-of-ai-in-education/

United Nations Human Rights Council. (2024). Contemporary forms of racism, racial discrimination, xenophobia and related intolerance (A/HRC/56/68). https://documents-dds-ny.un.org/doc/UNDOC/GEN/G22/336/38/PDF/G2233638.pdf

Link to “Harmony Living: A Future of Balance and Bliss”

What theoretical underpinnings are evident in our textual architectures and how does this affect one’s experience of the work?

Comparing Jaime’s “Harmony Living” with my own “Flippancy Pill” reveals interesting differences in how we approached the speculative futures assignment. We each made distinct choices about organizing our ideas that reflect different—but equally valid—ways of thinking about technology and society.

Jaime created a thoughtful two-part structure in her work. She first presents Neo-Terra, a beautifully detailed world where technology has enabled social harmony. Her vivid descriptions of “symmetrical yet soft” structures and AI tutors that personalize education create an immersive experience. Then she shifts to analysis, raising important questions about labor displacement and who defines what’s “preferable.” I appreciate how this approach gives readers space to first experience the emotional appeal of technological solutions before considering their potential problems.

Her organization shows a clear commitment to both creative storytelling and critical analysis. By separating these elements, she creates a reading experience that mirrors how we often engage with new technologies in real life—initial excitement followed by deeper consideration of consequences. Her references to Allen, Harari, and Dunne and Raby effectively connect her fictional world to current scholarly conversations.

My “Flippancy Pill” takes a different organizational approach. While writing, I discovered I couldn’t separate the narrative from the critique—they became fundamentally intertwined. This structural choice reflects my interest in how our theoretical assumptions shape our technological interventions. When I described the pill making it “impossible to retreat to the safety of ego-stroking frameworks,” I was trying to blur the line between describing a technology and analyzing its implications.

Through developing this speculative technology, I found myself demonstrating how theoretical assumptions can be embedded within the very design of speculative technologies. The pill itself embodies a philosophical proposition: that people adopt political frameworks based on what intuitively appeals to them rather than objective evidence. The pill functions as a kind of “truth-serum” that makes it impossible to see that your own viewpoint is based on any higher truth. It doesn’t directly make you appreciate opposing viewpoints, but strips away the pretense that your own position—whether social constructionist or naturalist—is grounded in anything more objective than personal preference.

An important critique of this concept asks: “What about those who run an authentic risk of being ostracized for merely thinking about a view that would go against their self-imposed status quo?” This raises fascinating questions about what kind of social breakdown would paradoxically result if people lost their conviction—biased though it may be—in their ideological positions. Group solidarity dissolves because the pill undermines the shared pretense that often forms the basis of group cohesion. The pill relates to the “steel man argument,” though indirectly. While it doesn’t directly make people steel man opposing positions, it creates necessary conditions for such engagement. In the case of a leftist who believes in social constructionism, they would find and elaborate something that seems indubitably “natural” rather than reflexively attributing everything to socialization. The pill makes you confront the most compelling aspects of opposing viewpoints in all their glorious arbitrariness, while simultaneously revealing the arbitrary nature of your own position.

These different choices create distinct reading experiences. Jaime’s clear separation gives readers an organized path through both storytelling and analysis. My approach asks readers to grapple with philosophical questions that arise directly from the speculative concept itself. Both approaches have their benefits—clear structure makes complex ideas accessible, while integration raises immediate questions about what happens when someone faces social ostracism for abandoning their ideological tribe, or how temporary such radical honesty might be.

Neither approach is more innovative than the other. Jaime’s careful separation allowed for both rich world-building and focused critique, while my integration of concept and critique served different purposes. These different approaches simply reflect the range of ways we can think about technological futures—each valuable for what it offers readers.

The contrast between our works demonstrates how the same assignment can generate complementary perspectives, with each contributing something unique to our collective understanding of technology’s role in shaping possible futures.

Link to “Task 10 – Attention Economy”

Isabella’s reflection on the User Inyterface assignment offers an interesting contrast to mine.  While I approached the interface primarily as a collection of technical problems to be identified and solved, Isabella framed her experience through accessibility concerns, particularly how her ADHD affected her interaction with the deliberately confounding design.

Her description of the countdown timer creating “genuine anxiety” and her comparison to an “escape room” highlight dimensions I hadn’t considered in my analysis.  Where I cataloged the misleading buttons and confusing checkbox placements with a certain detached frustration, Isabella articulated how these elements created actual barriers rather than mere annoyances.

What we both recognized, however, was the deliberate nature of these design choices.  Isabella noted how the site was “intentionally designed to manipulate” user responses, which parallels my observation about interfaces that make users “sign away their lives” through seemingly simple interactions.

Perhaps most revealing is our different responses: I stubbornly pushed through despite frustration with the interface, while Isabella made the reasonable choice to walk away.  The difference makes me wonder how many people simply can’t access services because of these awful design practices.

Link to “Task 11: Option 2 Text-to-Image”

In Elaine’s post, I noticed key limitations with Copilot’s image generation that didn’t come up during my fusion cuisine tests.  When she tried to get Copilot to create a visual of S&P 500 historical data, it failed entirely – revealing that these tools simply can’t handle factual data representation, no matter how clearly you ask.

Her results provide an interesting contrast to my own work.  While I was exploring creative food combinations, Elaine’s post shows something more fundamental about these image generators – they can’t function as search engines or data visualization tools, even for straightforward requests.

I think what’s happening with that S&P chart attempt is we’re seeing the outer edges of what an image generator can do when asked to be a search tool.  Reading the writer’s thoughts, I totally agree when they say: “It is doubtful if AI generates image from capturing key words from instruction instead of thinking the logic behind, the purpose of the instruction but rather mechanical.”  Then she tried to get an image for her twins and it came out with an image for 3 boys, so it wasn’t even good at analyzing specific words.  Similarly, in my fusion food prompts, the AI grabbed onto obvious terms like “sushi” and “Mexican” but missed how these should actually work together – revealing a shallow pattern-matching incapable of grasping the deeper logic or purpose.

I was surprised by their discovery about word order – putting “1 year old” before versus after “twin boys” completely changed what the AI produced.  When I was experimenting with food imagery, I didn’t test how word sequence might affect my results.

I found it interesting that we both independently discovered the same keyword fixation issue despite working with totally different content types.

Link to “Task 9: Networking Assignment Using Palladio & Golden Record Quiz Data”

Thomas raises some interesting questions about the factors that might shape our music selections: educational background, culture, and so on. These dimensions certainly influence how we engage with music.

Looking at my own experience with the network visualization compared to Thomas’s, I noticed something peculiar. I ended up being somewhat of an outlier, but the reason for this was simple and at the same time had very interesting implications. My position as an outlier had little to do with unique musical taste; it had to do with missing the YouTube reference page with the standardized names. Had I based my curation assignment on that page, I would have understood that I was expected to choose 10 songs, and not the spoken word pieces (one of which incorporated whale sounds).

This realization connects to Thomas’s discussion about similar choices. What appears in the visualization as meaningful difference in preferences might actually reflect nothing more than procedural differences in how we approached the task. When we extend this insight beyond our classroom, we can see how what looks like preference or choice in data visualizations might actually reflect unequal access to information or resources. Procedural differences in data collection often mirror deeper systemic barriers that divide populations along lines of privilege and access. In this way, marginalization can be reinforced by data. You may be in a position where you are representing a privileged interpretation—but it may not just be an interpretation, it may be an actual systematic, structural exclusion.

In educational settings like those Thomas mentions, we are often encouraged to use the data, but in doing that it’s important for us to consider the upstream factors that could lead to a certain “skewing” of the data. This awareness is crucial if we want to use these visualization tools in ways that reveal rather than reinforce existing power dynamics in learning environments.

Link to “Task 4: Potato Printing”

Patrick’s culinary skills really came through for this task. 10 minutes to do the cutting work? Only?! I laboured for two hours. He deliberately went with block letters and capitalization, while I was naturally drawn to curvature; out of some preconceived notion that simulating antiquity necessitated waves and flourishes? Pesky random flecks kept appearing in what should have been clean exterior spaces in my prints. Meanwhile, Patrick mentioned challenges with paint application and consistency. I kind of like the texture of his prints, the way the paint pools at certain edges of the letter and thins out elsewhere.

Patrick approached the assignment viewing each step as a hurdle to overcome and seemed fine with the results not being perfect. But I was not quite so easy-going, perhaps because I had put in so much elbow grease. Luckily, my wife brilliantly suggested stamping each letter twice, as the shorter interval would keep the muscle memory fresh. This makes me wonder: did Patrick go the proper route, stamping the whole word out before attempting the new word?

Patrick wished he had done the task with his son – my kids were interested, but seeing the tedium, elected not to participate. They were, however, thrilled with the finished product. The stamping was a lot of fun for them. Patrick also spoke approvingly of how printing enables efficient word dissemination. I was also appreciative of how much easier printing is than letter production by hand. The rigor of what they went through in the past is truly astonishing.

Link to “Task 3: Voice to Text Task”

Both Justine’s post and my post focused on how transcription captures the unique qualities of storytelling – features that would ideally, if not always, be smoothed out in written text. I paid particular attention to things gone awry such as structure and tense shifts; Justine focused on fillers, repetition and the like. Both of us, I guess, were somewhat disparaging of speech. Why is it that writing should be treated any more formally?

But this makes me wonder: why does writing have a privilege over speaking? It’s my understanding that Derrida has written many books about this. So maybe I was completely wrong to suggest that these are deviations, because it’s clear that the written word has long since enjoyed privilege of place.