Text Technologies: Textual Analysis for Mental Health an Academic Success
In recent years, the link between student mental health and academic success has become a prevalent topic of discussion on postsecondary campuses, and for good reason. In their policy paper exploring mental health issues on Canadian campuses, Max & Walters (2018) note that according to The Mental Health Commission of Canada (MHCC) , “the highest rate of mental health problems and illnesses is among young adults ages 20 to 29, a time when young people are generally beginning post-secondary education and careers” and that “About half of post-secondary students with mental health disabilities will experience the onset of their condition over the course of their post-secondary education.” (p. 3). Given that this is the case, we shouldn’t be surprised to learn that a 2016 survey of Canadian post-secondary students indicated that 44.4% of respondents reported feeling “so depressed it was difficult to function” (Max & Walters, 2018). Students with depression and anxiety are also more likely to experience academic challenges, lower grades and are more likely to drop out of school compared to their peers who do not report mental health challenges. (Eisenberg et al., 2009). Taken as a whole, the research is clear: mental health is increasingly becoming a central focus of the student experience, and its associated impacts on key indicators of University success- academic achievement, advancement and retention, are significant.
Recent advancements in educational technology provide use with new tools to identify students who are at risk of developing mental health issues, failing their classes or abandoning their studies altogether. Traditionally, educators would look for early warning signs by observing student behavior- class attendance, personal hygiene, and overall engagement in course material. With online learning become more prevalent in recent years, these traditional metrics are becoming more difficult to observe. In many classes, course content is delivered not by lecture but via online discussions where students are welcome to engage with the course at their own pace. While this new format may make it more difficult to detect early warning signs based on behavior, it does provide an opportunity to shift our focus to the actual written content that students are submitting. We can learn a great deal about a student’s mental state by examining their writing. This is a form of what is called content-oriented analysis- where we analyze a collection of writings, either manually or with the assistance of technology. By analyzing the writing we can identify key themes and track changes in mood or behavior which can in turn be used to predict future outcomes including academic success or failure and mental health challenges.
Research on this topic has already taken place in a non-academic context, with promising results. In 2019, researchers in Czechia sought to explore the relationship between linguistic characteristics of an author’s text with their emotional state, specifically for the purpose of identifying subjects at risk for depression. Participants were asked to write sample texts which were then evaluated using a number of metrics including number words per sentence, use of punctuation, sentence complexity, word choice and length (or brevity) of writing. Using these methods, researchers were able to establish predictive models which were reliably able to detect individuals at risk of developing depression (Havigerová et al., 2019).
Similar methods have been employed in the context of post-secondary education in an effort to detect early warning signs of academic struggles among student populations. A multi-year study using data retrieved from Michigan Virtual School from 2014 through 2016 sought to employ textual analysis techniques on writing samples from class discussion forums, in an attempt to identify predictors of early warning indicators for detecting learners at risk of failure. The study was already using other data points- student behavior such as login frequency and frequency of course access, as well as demographic data such as age and gender. The study showed when text analysis data was included, it improved overall predictive accuracy. (Hung & Rice, 2018).
While multiple data analysis techniques were used in the study, one I’d like to emphasize is the Linguistic Inquiry and Word Count (LIWC) word analytic tool. The LIWC is a tool that focuses on using text analysis to reveal how students are thinking, as opposed to the content of what they write. The method by which this is achieved is by counting the use of function words in writing samples. A function word is one with a grammatical purpose- this includes pronouns, determiners and conjunctions. These function words can be useful indicators of the author’s psychological state. Research shows that text that features heavy usage of prepositions and articles indicates higher levels of cognitive complexity, a writing style referred to as “categorical”. Conversely, text that employs heavier usage of pronouns, auxiliary verbs, adverbs, conjunctions, impersonal pronouns, personal pronouns, and negations indicates a more “dynamic” writing style. Higher academic performance has been shown to correlate with categorical writing styles. (Pennebaker, et al, 2014).
In this case, while employing the LIWC tool in their data analysis, researchers limited their focus to eight key dimensions: word count, analytic, authentic, words per sentence (WPS), six letter words, function words, pronouns, and cognitive process. When the results of the analysis were added to the existing data collected, they found that successful students used more words that were determined by the algorithm to be “analytic”, whereas the at-risk students used more words that were deemed “authentic”, such as personal pronouns. An argument could be made that the at-risk student group were more focused on themselves in their writings, whereas the successful student group was more focused on course content.
The results of the study of student writing samples at Michigan Virtual School are illuminating, and if the information collected is acted on appropriately, it could prove to be a useful tool for early identification and subsequent intervention of students who are at risk for failure, or are in need of mental health support. However there is one potential issue with using the data this way- it’s very difficult to determine to what degree potential bias coded into the LIWC analysis tool may be skewing the results. The trouble in this scenario is that the tool was programed by humans to begin with, so inevitably their own biases will be present in the algorithm. Furthermore, a raw text analysis is completely devoid of context. While some biographical data was collected as part of the study, there’s no indication that the LIWC results were cross-referenced with that data. Perhaps some cultures, genders or age cohorts are more likely to employ analytic word choices in a more categorical writing style. Furthermore, it’s unclear by what metric the LIWC tool determines which words are analytic versus authentic. The literature does include general guidelines, yes, but human language is very complex. Depending on an individual’s writing style, the context they use their choice words in could very much determine how it should be categorized. Pronoun usage many indicate a focus on self during the writing process, however if pronouns are relevant to the course material at hand, they may just as likely indicate a focus on course content, which is associated with better learning outcomes which in turn correlates with better mental health.
In addition to bias, we should there are potential ethical concerns to take under consideration. Use of student data is a bit of a contentious issue presently, as is use of personal data by any institution. Certainly the data collected through text analysis techniques could be highly beneficial to the student, the course instructor and the institution at large. Identifying students who are at risk of failure or mental health challenges could result in a stronger network of student support, but if not collected or used ethically could also be seen as an invasion of privacy. In order to ensure this isn’t the case, any institution engaging on large scale data collection should first ensure it has a policy in place pertaining to the use of student data derived from analytic platforms. An industry leader in this regard is The University of West London, which drafted a progressive learning analytics policy in 2016. The policy is student centered in that its stated purpose is to help students with their academic pursuits. The policy includes ten core principles for ethical use of collected data. Among these include consent (students must explicitly agree to having their data collected), openness (use of data must be transparent to all stakeholders) and individuals (students are individuals and will not be wholly defined by their data). In today’s world, it’s imperative institutions to draft similar policy with regards to their data collection practices. Students should be consulted at every step of the way, and the policy should be made clear and available to the entire campus community. A well drafted policy won’t guarantee that student data isn’t used inappropriately, but it will increase confidence in the institution and serve as a strong foundation for learning analytics projects and research, like the one at Michigan Virtual School.
The practice of writing has been evolving constantly since its inception many thousands of years ago. The emergence of computers as a writing tool, along with the rise of the World Wide Web in the late 20th century are perhaps the most recent paradigm shifts which change our approach not only to how we write, but how we consume written content. With recent developments in learning analytics, we now have the capability to use technology to pull additional meaning from writing. In addition to the content of writing, we can use analytical tools and frameworks to determine the state of mind of the author, and make reasonable inferences with regards to their future behaviors. With the link between mental health and academic success clearly established, institutions have a powerful tool at their disposal to offer support and guidance to at risk students. Ethical use of these tools could prove to be very beneficial not just to the institution but to the students and campus community as well.
Works Cited
Brett, Megan R. “Topic Modeling: A Basic Introduction”. Journal of Digital Humanities. Retrieved from http://journalofdigitalhumanities.org/2-1/topic-modeling-a-basic-introduction-by-megan-r-brett/
Chen, Ye & Yu, Bei & Zhang, Xuewei & Yu, Yihan. (2016). Topic modeling for evaluating students’ reflective writing: a case study of pre-service teachers’ journals. 1-5. 10.1145/2883851.2883951.
Eisenberg, D., Golberstein, E., & Hunt, J. B. (2009). Mental health and academic success in college. The B.E. Journal of Economic Analysis & Policy, 9(1), 40. doi:10.2202/1935-1682.2191
Havigerová, J., Haviger, J., Kučera, D., & Hoffmannová, P. (2019) Text-based detection of the risk of depression. Frontiers in Psychology. doi.org/10.3389/fpsyg.2019.00513
Hung, A & Rice, K. (2018). Combining data and text mining to develop an early warning system using a deep learning approach. Lansing, MI: Michigan Virtual University. Retrieved from https://www.mvlri.org/research/publications/combining-data-and-text-mining-to-develop-an-early-warning-system-using-a-deep-learning-approach/
Max, A., & Walters, R. (2018) Breaking Down Barriers: Mental Health and Canadian Post-Secondary Students. Retrieved December 1 2020, from https://www.casa-acae.com/breaking_down_barriers_mental_health_and_post_secondary_students
Pennebaker, J. W., Chung, C. K., Frazee, J., Lavergne, G. M., & Beaver, D. I. (2014). When Small Words Foretell Academic Success: The Case of College Admissions Essays. PLoS ONE, 9(12), 1-10. doi:10.1371/journal.pone.0115844
University of West London. (n.d.). In Wikipedia. Retrieved August 4th 2022, from https://en.wikipedia.org/wiki/University_of_West_London
Vaidya, N. (n.d.) 5 Natural language processing techniques for extracting information. Retrieved from https://blog.aureusanalytics.com/blog/5-natural-language-processing-techniques-for-extracting-information