Introduction
Text has been evolving alongside technology throughout human history. As humanity develop and utilize the scroll, the codex, the printing press, hyperlink text, and predictive text of algorithms, aspects of text and human thought changes. In an age of gamification, instant feedback, and advancements in machine learning, one possible next step of text related technology would be automatic writing assessors (AWAs). Rather than modern tools such as spellcheck, autocorrect, and predictive text which provides feedback as one writes, AWAs evaluate and provide feedback on an entire piece of writing rather than words and sentences, allowing the user to act on that feedback and refine their writing.
Various computer algorithms already exist that associate variables of a piece of writing to its quality. For instance, the free online scientific journal depository mentioned in module 1 of this course, arXiv, utilizes an algorithm that detects the presence of key phrases updated through machine learning to determine if an article is scientific and therefore should be added to arXiv (Becker 2016). AWAs can function in the same manner, though instead of an output of ‘scientific, therefore, should upload to arXiv’ or ‘not scientific, therefore, should not upload to arXiv,’ AWAs can assign a numerical score to a piece of writing and/or provide feedback on the piece of writing.
In this article I will discuss my pedagogical context, the history of AWAs, how a machine learning technique, random forest, can train AWAs, how AWAs can be used in my context, and concerns that may rise from the development of AWAs.
Personal Pedagogical Context
I began my education career as a science teacher, and a few years ago I began to teach an introductory epistemology class, Theory of Knowledge (TOK), for the International Baccalaureate (IB) Diploma Program. Unlike assessments for science classes which typically contain questions that require students to follow various step to solve and I can determine students’ thinking process depending on the work shown, TOK assessments are various pieces of writing that I submit to the IB organization for moderation. For the TOK Exhibition where students have to select three objects and relate them to one of the prompts provided by IB in a maximum 950-words piece of writing. The TOK Exhibition is a third of a student’s overall TOK mark, and after grading all of my students’ exhibitions IB randomly selects certain students to ensure that I’m neither too hard or too easy with my marking. In my first years teaching TOK, IB determined that for the students I rated highly, I was too generous, while for students from the mid to low range I matched their assessment practices. IB TOK rules also stipulate that to ensure that the exhibitions are the students’ own work, as a teacher I’m only allowed to read and provide feedback on one draft. All these factors make me wish for a tool that a) allows students to check and refine their own work without teacher input, and b) allows me to better grade these exhibitions to match IB standards. This is where AWAs could be of use.
Evolution of AWAs
According to Shermis & Burstein’s Handbook of Automated Essay Evaluation: Current Applications and New Directions (2013), the concept of AWAs stemmed from Ellis Page’s 1966 article, The Imminence of… Grading Essays by Computer. In 1993 Wresch referred back to Page’s article and concluded that the concept was still being explored with nothing momentous yet. Fast forward to 2012 when the Hewlett Foundation organized a contest to evaluate AWA tools and found that the quadratic weighted kappa (QWK) rates, a statistics metric used to determine whether two scores agree, for human scorers on the samples provided were between 0.75 to 0.97 and the winning AWA tool had a QWK of 0.77 (Shermis & Wilson 2024). In the past decade, AWAs have been developed further, and in 2021-2022, the US’s National Assessment of Educational Progress (NAEP) program ran a contest on AWAs and found that the human QWK was 0.91 while the winners had QWKs of 0.89, 0.88, and 0.87 (Shermis & Wilson 2024). AWAs are being widely adapted, with several school districts in the United States such as “South Dakota, Utah, North Carolina, Louisiana, Ohio, and West Virginia” using AWAs to provide formative feedback to students (Shermis & Wilson 2024).
Various AWAs are currently on the market, such as Pearson’s Intelligent Essay Assessor (IEA) that “uses machine learning and natural language processing to score essays and short answers in the same way human scorers do” (Pearson n.d.), Page’s Project Essay Grade (PEG) where “students who received PEG feedback alongside traditional writing instruction demonstrated a 22% stronger improvement in their writing skills compared with those who did not” (ERB Team 2023), and Vantage Labs’ IntelliMetric where a Californian school district “was able to evaluate student writing and their students were able to use the instantaneous feedback to drastically improve their writing. The majority of teachers found IntelliMetric to benefit their classrooms as an instructional tool and found that students were more motivated to write” (n.d.).
Rather than analyzing and evaluating these various tools on the market, the goal of this article is to provide my own proof of concept as to how an AWA can be developed using machine learning algorithms and how it could be used in my practice.
Machine Learning: Random Forest
As discussed in module 11 of this course, language-based machine learning algorithms are trained using a corpus of texts. Machine learning algorithms can statistically analyze its corpus to determine the likelihood of words that follow each other, allowing language based artificial intelligence to produce an output that passes the Turing test (Hall n.d.).
There are numerous types of machine learning algorithms such as neural networks, linear regression, logistic regression, clustering, decision trees, and random forest (IBM Cloud Education n.d.). Of these, the random forest caught my eye as it is more transparent rather than the “black box of the neural network” (IBM Cloud Education n.d.) and allows for internal evaluations of accuracy.
A random forest uses numerous distinct decision trees (IBM Cloud Education n.d.) to overcome issues with just one decision tree such as overfitting (Yee & Chu 2015). A decision tree uses a series of machine generated variables to split the data into smaller groups to predict something (Yee & Chu 2015). For instance, variables to determine the strength of a piece of writing could include the number of key phrases used, the word count, the number of commas and/or periods, and the average word length. A decision tree for an AWA could first look at the word count and determine that anything above the maximum or below the minimum won’t make the cut. Then, it could look at the presence of key phrases to predict if it’s on topic, as well as counting the number of words between periods to determine if there are any overrun sentences. Sometimes, overfitting occurs, where variables selected for a decision tree are irrelevant but exist in the tree because they match the training data (Yee & Chu 2015). The random forest overcomes the overfitting issue though the generation and evaluation of numerous decision trees.
Starmer 2018 provides an introductory glance of how random forests work. To apply the concept to AWAs, imagine that we are given a sample of 100 TOK Exhibitions, all already assessed by IB. The first decision tree randomly selects 72 of these exhibitions and uses variables such as presence of key phrases, average word length, and word count to match each of the exhibitions to their scores. The second decision tree selects 68 exhibitions and uses variables such as the number of commas, the absence of words associated with a bad exhibition, and the word count. This process repeats until there are numerous distinct decision trees that make up a random forest. For each tree, the unused samples can be used to determine the accuracy for a tree. For the second tree, if the decision tree scores the 32 unused samples of the training data inaccurately, it can be pruned from the forest (perhaps the number of commas was an overfitting variable).
Once the forest is generated through training data, it can then be used to evaluate new data. If 30 decision trees are generated and 27 of them thinks an exhibition should receive 7/10 while 2 of them evaluates it at an 8 and the final one evaluates it at a 6, that exhibition should receive 7/10.
Pedagogical Considerations
Note that while the variables listed above are just examples and may have not a strong correlation with the quality of a piece of writing. Through the examples of arXiv and the NAEP contest, AWA tools currently can be used to gauge whether or not a piece of writing belongs in a certain category (whether something is scientific or should have a certain score). While currently language-based algorithm models do not operate off of understanding the criteria of a rubric but statistical likelihoods, it can still provide a similar outcome as a human. In the context of the IB TOK Exhibition where teachers can only provide feedback on one draft, this provides students with another potential tool to evaluate and refine their work. This could also provide instructors another guide in scoring TOK Exhibition, if IB scores (rather than a specific teacher such as myself) were used for the training data.
In a way, an AWA for the IB TOK Exhibition using random forest is already similar to how I introduce the exhibition to my students. One activity I do is to have students go over the rubric, then have them go around to various stations to read sample exhibitions provided by IB and rate them. After we compare what score they give the exhibitions to the actual IB scores, students then return to each exhibition to study various features of the samples that led to their score. While hopefully students don’t come up with something more frivolous such as the number of commas used, strong students were able to gain an understanding of what a highly scored exhibition requires. After generating a piece of writing, the cycle of having it evaluated by an accurate AWA and refining a piece of writing can level the playing field and help all students understand the aspects and requirements of a strong exhibition, especially in this context where continuous teacher feedback is forbidden.
Concerns
AWAs can spark various concerns that parallel those of algorithms that we have discussed in module 11 of this course around training data for language-based algorithms: the privacy concerns exemplified by the Enron emails mentioned by Herman 2019 and how various algorithm using tools perpetuate preexisting biases (Talks at Google 2016). Recently there has also been concerns about how machine learning using copyright material as training data, exemplified by a lawsuit by New York Times against OpenAI mentioned by Reed 2024. These issues would be minimized with a mindful approach to AMAs. The privacy and copyright issues of the training data can be mitigated by asking for student agreement to provide their works as training data and removing any personal information from students’ work prior to using it as training data. While it’s true that AMAs use would perpetuate preexisting biases of the original examiner(s) who assessed the training data, the assessment process that leads to the training data and the random forest method of machine learning is transparent and allows for policy makers of various educational institutions such as IB to audit to determine the presence of any unfair bias.
Conclusion
Recent advancements in text technology such as spellcheck, predictive text, and even language-model artificial intelligence such as ChatGPT have reduced the production of text. Instead, these technologies encourage the selection of machine generated suggestions. AWAs would allow a shift back towards text production, providing instantaneous feedback on a piece of writing created by the user.
References
Becker, K. (2016, October 13). What Counts as Science. Nautilus. https://nautil.us/what-counts-as-science-236150/?_sp=eec044e8-5f5a-44bb-af98-6d4cab8e3a2b.1722903148511
ERB Team. (2023, August 10). How the AI-Driven PEG Scoring Algorithm Can Improve Student Writing. ERB. https://www.erblearn.org/blog/peg-scoring-algorithm/
Foltz, P. W., Streeter, L. A., Lochbaum, K. E., & Landauer, T. K. (2013). Implementation and applications of the intelligent essay assessor. In M. D. Shermis, M. D. Shermis, J. Burstein & J. Burstein (Eds.), Handbook of automated essay evaluation (1st ed., pp. 68-88). Routledge. https://doi.org/10.4324/9780203122761-5
Hall, D. (n.d.). 99% Invisible (No. 382). Retrieved December 12, 2022, from https://99percentinvisible.org/
Herman, C. (Host). (2019, June 5). You’ve got Enron mail! (no. 35). [Audio podcast episode]. In Brought to You By. Business Insider.
IBM Cloud Education. (n.d.). What is machine learning (ML). IBM. https://www.ibm.com/cloud/learn/machine-learning
Page, E. B. (1966). The imminence of… grading essays by computer. Phi Delta Kappan, 47(5), 238-243.
Pearson. (n.d.). Automated Scoring. Pearson Assessments. https://www.pearsonassessments.com/large-scale-assessments/k-12-large-scale-assessments/automated-scoring.html
Shermis, M. D. & Burstein, J. (2013). Handbook on automated essay evaluation: Current applications and new directions (1st ed.). Routledge. https://doi.org/10.4324/9780203122761
Shermis, M. D. & Wilson, J. (2024). The routledge international handbook of automated essay evaluation. Routledge.
Starmer, J. (2018, February 5). StatQuest: Random Forests Part 1 – Building, Using and Evaluating. [Video]. Youtube. https://www.youtube.com/watch?v=J4Wdy0Wc_xQ&ab_channel=StatQuestwithJoshStarmer
Talks at Google. (2016, November 2). Weapons of math destruction | Cathy O’Neil | Talks at Google.
Vantage Labs. (n.d.). Intellimetric. https://www.intellimetric.com/
Wresch, W. (1993). The imminence of grading essays by computer—25 years later. Computers and Composition, 10(2), 45-58. https://doi.org/10.1016/S8755-4615(05)80058-1
Yee, S. & Chu, T. (2015, July 27). A visual introduction to machine learning. r2d3. http://www.r2d3.us/visual-intro-to-machine-learning-part-1/