Scenario 1.

Data overdrive: How much data is too much data?

Subject: what do you think about the new harvest helmets?

Friends,

I know a lot of us just accept that school is the way it is, but have you ever thought of all the ways we’re being watched? Like really thought about it? 

My mum says smart watches and eye tracking were around even before her time; ancient holdover technology from the early part of this century. I suppose schools make us use them because they’re so cheap and because so much of the current school budget has been used to purchase Harvest Helmets from Datum Fortis. The way I understand it, our smart watches are synced with the schools Central Data Gathering Authority (CDGA) and send information about our sleep cycles, activity levels, heart rate, and body temperature. There are rumours too, that the watches also send information about how often we go to the bathroom and when we’re on our periods, but I can’t say for sure because the data collection has never really been explained to us other than the school motto “gather, manage, monitor.” As for eye tracking, all of our devices are equipped with software that simultaneously tracks what apps, websites, and virtual worlds we participate in, as well as how long we interact with these spaces and where our attention is spent. If we choose to write by hand, our digital pens send data to the CDGA, including how often we pick up our pens and put them down, how long we spend writing, and how often we erase or cross out our work. It also analyzes our pen grip as a metric for stress, similarly, our digital paper is pressure sensitive for this same reason, did you even know that? I haven’t even touched on the fact that anything we produce as students is collected, coded, sorted, aggregated, and analyzed, but this is such a common practice that I don’t think students even acknowledge it’s happening.  The latest technology to come out of Datum Fortis, the for-profit leader in ed tech, is the harvest helmet, since (apparently) our smart watches, pens, and computer monitoring software are not enough. The Datum Fortis harvest helmets collect our brain activity using proprietary cerebrum interference technology and we’re expected to wear them at minimum for the hours we’re engaged in school activity, but we’re strongly encouraged to wear them for as long as we can tolerate having them on. I think the CDGA, ideally, wants to be able to data-mine our brain activity on non-school activities as well. The thing is, no one tells us anything about how all this data collection and mining works and to what end. I’m not even certain that this technology is one way, how do I know if the helmet is feeding me information or altering my brain in some way? How can I be certain where my data is going? How long is the data stored for? If you also want these questions answered, sign the petition for the “freedom of discovery” request I’ve attached to this email. 

Unity and mutiny,

Daphne

P.S. Did you know that Datum Fortis is a subsidiary of the media conglomerate DisNet?

Scenario 2.

Living tissue masks: a breath of fresh air

The Covid-19 pandemic has certainly brought to the forefront of people’s mind the realities of communicable diseases and the vectors that spread them. As the climate continues to change and warm, we can expect the incidences of known (and possibly unknown) communicable disease to rise. Simultaneously, climate change can also negatively impact local air quality. 

As people become accustomed to wearing masks it makes sense to leverage this new norm to tackle the dual problems of airborne disease and poor air quality. At Studio Plante we have conceptualized and prototyped the living tissue mask. Taking inspiration from nature, the idea for the living tissue mask capitalizes on the abilities of plants to purify surrounding air.  Research has shown that plants like Epiprenum aurum, aka pothos ivy, use enzymes to remove toxic chemicals from the air, and scientists have already been able to boost this ability by splicing in the animal 2E1 gene, which expresses enzymatic activity. One challenge to developing the living tissue mask is keeping it alive, and pothos has a root system that requires submersion in water or soil. To overcome this challenge, the already genetically modified pothos plant, can be spliced further with genes from the genus of plants known as tillandsia, or more commonly, air plants. Pothandsia, as we have come to call it, retains the air purifying ability of the transgenic pothos, combined with tillandsia’s ability to live in low moisture conditions on a variety of substrates (including both living and non-living substrates). 

Our living tissue mask is made out of pothandsia grown on an organic muslin substrate and survives off the natural circulation of air from breathing, as well as the moisture content in the wearer’s breath. 

Users of the mask will find improved breathability, enhanced air quality, and a significant reduction in disease exposure. Our belief at Studio Plante is that health and climate are a community responsibility, and it is for this reason that we have made our research and design open access. 

Living Tissue mask prototypes

Reflection

When creating the speculative future scenarios, I considered a few ideas from Dunne and Raby’s (2013) book Speculative Everything:

• Don’t try to predict the future, instead create ideas that lead to discussion around the kinds of futures people really want 
• Start with a what if question 
• Keep scenarios scientifically possible
• Provide a clear path from where we are presently to the speculated future scenario
• Represent ideology and values
• Choose a purpose (critique, entertainment, catalyst for change, inspiration, reflection)

In Scenario 1, I wondered: what if data-driven education is taken to the extreme? This is not a prediction of where education is going, but a scenario that offers critique and reflection on what that world might look like. I imagined more of a dystopian future and hopefully the scenario brings to mind ideas around the ethics of data collection, such as surveillance, algorithms, and transparency. Additionally, perhaps it sparks questions such as, how much data is enough data? And who gets to access the data?  Maybe it brings to mind ideas of data storage, what do we keep, what can we afford to lose (Brown University, 2017)?

For scenario 2, I was moved by the idea in Speculative Everything (Dunne and Raby, 2013) of imagining how nature could be different. This scenario, rather than a ‘what if,’  takes the shape of a thought experiment. I considered a more idealistic speculative future where design isn’t driven by market forces, but one that represents values such as care for humanity. While thinking of this scenario, I was reminded of technology such as parchment which capitalized on animal tissue to create writing substrates that were durable and long lasting. Additionally, I tried to create a clear path for how we can take our current scientific understanding of genetically modified organisms to create a mask that could possibly be made out of living tissue. For example, pothos has been genetically modified with a gene for an enzyme from rabbit liver (Zhang et al., 2019) and it is true that air plants can survive on many types of substrates in relatively low moisture conditions (Tillandsia, 2021). As for whether or not pothos could be further genetically modified with genes from air plants, that has yet to happen, but seems plausible. 

One thing that I didn’t mention in my scenario, but did think about with respect to education, is that indoor air quality is often poorer than outdoor air quality, and if students are required to wear masks in school, why not use masks that purify the air?

References

Brown University (2017, July 11). Abbey Smith Rumsey: Digital memory: What can we afford to lose [Video]. YouTube. https://www.youtube.com/watch?v=FBrahqg9ZMc.

Dunne, A., & Raby, F. (2013). Speculative Everything: Design, Fiction, and Social Dreaming. Cambridge: The MIT Press.

Tillandsia (2021, March 4). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Tillandsia&oldid=1010258005

Zhang, L., Routsong, R., & Strand, S. E. (2019). Greatly enhanced removal of volatile organic carcinogens by a genetically modified houseplant, pothos ivy (epipremnum aureum) expressing the mammalian cytochrome P450 2e1 gene. Environmental Science & Technology, 53(1), 325-331. https://doi.org/10.1021/acs.est.8b04811

My predictive text and I have something very important to say:

As a society, we are in a good place to make a good point. The only problem is that the game doesn’t play with the other games and I can’t stop playing it. 

Why does my predictive text sound like a dude bro in his first year of a philosophy major? I truly hope that outside of my predictive text, I don’t sound like someone who is stereotypically high waxing poetic, but never actually saying anything of substance (that would be mortifying).  Is this exemplary of how I write? Is predictive text a reflection of who we are? In some ways, the answer is both yes and no. Predictive text uses machine learning in order to assess which words we tend to use more often and creates a personalized dictionary that scores those words based on the probability that we’ll use them again (NBC News, 2017). Additionally, the predictive text algorithm also uses something called “probabilistic language modelling” which considers the context of what is being written and how certain words tend to go together (NBC News, 2017). Ultimately, the algorithm of predictive text is a combination of machine learning (about how I speak) and language probability (how everyone speaks). This leads me to wonder, how much of the predictive text is me and how much of the predictive text is language modelling?

Interestingly, some predictive text, such as Smart Compose in Gmail is based off of a finite corpus of emails from Enron, (though I’m unclear if all predictive text uses this base of emails for developing its algorithm) (Mars, 2020). As Amanda Levendowski points out, It doesn’t take a lot to imagine how basing predictive text and other AI off of a specific set of emails might bias the technology (Mars, 2020). Is it possible that the probabilistic language modelling of my phone’s predictive text algorithm is based off of the emails of Enron dude bros embroiled in an early aughts corporate scandal?

In some ways using predictive text feels like using a Ouija board with your friends when you’re eleven years old . You may think it’s random or believe that it’s ghosts, but in reality you and your friends exert force onto the planchette of the Ouija to make it spell something coherent (Romano, 2018).  Similarly, predictive text is not random, nor is predictive text a ghost writer, we guide the predictive text by the choices we make and it coughs out a sentence. 

I tried the predictive text a few times with a few different prompts, partly because it was so fun, but also to see if I could get a result that I thought sounded like me. Interestingly, I found the predictive text leaned more toward a positive affect, for example the words ‘good’ and ‘great’ came up quite often (as well as the word ‘birthday’). Every time I put in the prompt “as a society, we are…” I got sentences like “we are going to be able to make it”  or “we are in a good mood” or “we are in a good place.” Interestingly, I don’t think I would ever finish that prompt that way. I’m quite critical of society actually, but the predictive text never gave me the option!

I also found that the predictive text never really led to saying anything quite meaningful, consider this sentence generated by predictive text: I think we should have some more of the things we need to make sure we are doing the next year. It reminds me of a segment of John Oliver’s Last Week Tonight in which John Oliver compares Trump’s actual presidential speech patterns with that of predictive technology (Last Week Tonight, 2017,  3:10).

As an extension to this activity, I also tried using predictive text for something that I wanted to say (rather than using a provided prompt and seeing where the predictive text meanders). In this example I’ve highlighted the words that the predictive text offered that were indeed predictive of what I was intending to write (I did not include the words that the predictive text offered after I had typed a few letters of the word):

You should check out the episode “you’ve got mail” from 99% invisible which is about how basically all AI technology, like predictive text, is based on the released emails from Enron. 

Certainly there is a difference between using predictive text to generate content versus using predictive text to support content. In a study about predictive text the authors found that when using it to caption photographs it led people to write shorter, more predictable captions (Arnold et al., 2020). When I think about predictive text in various contexts, I can see its value in situations that require more formulaic writing or in contexts where brevity is valued. Writing work emails would be an example of this type of context; there is a specific language and way that people conduct themselves in work emails. Anyone that has had to write or receive an email for work would be well acquainted with the phrases “please see the attached document,” or “at your earliest convenience,” and the dreaded “as per my last email.” However, in the end, I would make the case that predictive text works best when we use it to support our ideas, not generate them. 

References

Arnold, K., Chauncey, K., & Gajos, K. (2020). Predictive text encourages predictable writing. Paper presented at the 128-138. https://doi.org/10.1145/3377325.3377523

Last Week Tonight [LastWeekTonight]. (2017, November 13). The Trump presidency: Last Week Tonight with John Oliver (HBO) [Video]. YouTube. https://www.youtube.com/watch?v=1ZAPwfrtAFY

NBC News. (2017, November 8). Predicitve texting: How your phones keyboard figures out what you might type next [Video]. YouTube. https://www.youtube.com/watch?v=OfzMkERVFu8

Roman, M. (2020, November 9). You’ve got Enron mail (episode 421). [Audio podcast episode]. In 99% Invisible. Radiotopia. https://99percentinvisible.org/episode/youve-got-enron-mail/

Romano, A. (2018, September 6). How Ouija boards work. (Hint: It’s not ghosts.). Vox. https://www.vox.com/2016/10/29/13301590/how-ouija-boards-work-debunked-ideomotor-effect

Golden Record Curation Network generated by Palladio for ETEC 540

To be honest, I don’t have a strong understanding of algorithms and the terminology that I use in this reflection may make that quite obvious, but please bear with me. I do understand algorithms to be a set of instructions that help solve problems, but I wonder what problem is being solved with the Palladio program. When I look at the whole class graph/network it all seems pretty straight forward, the program has shown the connectivity of the class and their respective song choices for task 8. I could draw something like this by hand fairly easily by drawing a node for each song and each person, and then connecting them with a line. There wouldn’t be a need, necessarily, for any math in order for me to do this. Similarly, if I play around with the data and force groups of people, say for example forcing a grouping between Meipsy, Nathan, and Allison, I can see their connectivity and understand exactly how Palladio generated that graph.

Forced network grouping of Meipsy, Nathan, and Allsion

However, I wonder what set of instructions Palladio is using to group the members of each class. I would suspect it’s optimizing the in-group connectivity, but I’m not sure. Why does it put students into groups of 3 or 4? Why not two groups of 8 and one group of 7? Or why not groups of 4 or 5? Is this a default setting, is it part of the optimization, or was this predetermined by our instructor? 

The groupings generated by Palladio grouped James, Nathan, and myself together. As you can see in the image below, we have a high degree of connectivity with each other (with James and I being slightly more connected in the grouping than Nathan).

Palladio generated network grouping of Nathan, James, and Deirdre

Nathan has a good write up in his reflection about why he suspects we had such similar song choices. I don’t necessarily want to duplicate what he’s written, but I do agree that part of the reason is because the three of us created similar constraints for our song choices. Other similarities we share (those that I could glean from the info hidden in our blog posts) include the fact that we’re all high school teachers in math and science, and I’m assuming, generationally we’re all millennials. Though there are other people in our class who fall into those demographics (e.g. Ying)  and their song choices were not similar enough to be grouped with us. I suspect the constraints we determined for our song choices is a better reflection of grouping than anything about us demographically. But then again, maybe our demographics affected the constraints we decided on. Interestingly, Nathan and I actually know of each other having worked in the same remote region of Canada. Is there a shared personality trait that would lead us to both work in the Arctic and teach math and science and come up with the same choices of songs? I wonder how many other small world connections there might be in the other groupings. 

Had Nathan, James, or myself just chosen what we personally liked, would we have still ended up in the same group? 

Curiously, Palladio excluded Katrina from our group despite the fact that she shared 7 song choices with myself, 7 songs with James, and 6 with Nathan. I’ve included images of Katrina grouped according to Palladio and then grouped with my grouping.

Katrina’s original network grouping generated by Palladio

Katrina grouped with Nathan, James, and Deirdre

Interestingly, when visiting Katrina’s blogpost, she had similar constraints for herself when it came to song choices. Why then, did Palladio’s algorithm parse the data the way it did?

Another area of exploration is looking at who I shared the least amount of song choices with. In this case, Ben and myself only shared 2 song choices. I can’t speculate on why, given that I don’t have access to Ben’s reflections, but if my hypothesis is right, I would suspect that with respect to song choices, Ben had constraints that differed from mine.

Network depicting the connections between myself, Ben, and our song choices

One last thing I am curious about is if there are any two people in our class whose song choices excluded each other. I wasn’t able to determine an easy way to manipulate the information in Palladio in order to see that without manually examining each pairing of students.

Essentially, the issue at hand is whether or not Palladio’s groupings can accurately reflect anything more than just the network of connections of song choices, and the strength of the choices. Perhaps it would be helpful to have some insights into the biases or instructions used by the Palladio algorithm. Otherwise, I’m not convinced that a network like this can tell us anything about “why.” Why did our class overwhelmingly choose Johnny B. Goode and so few of us chose Men’s House Song?

This is the first time I’ve been exposed to Nasa’s Golden Record and I was struck with the thoughts that simultaneously it was so wholesome and so arrogant. The Voyager Golden Record seems wholesome in its quest to share the music of Earth and to find common ground among intelligent life, for example consider its inscription “to the makers of music – all worlds,  all times” (Taylor, 2019). However, it is also arrogant in its quest to represent humanity in a single LP, as well as arrogant to think any life in space would even care. It is true that in the podcast interview with Dallas Taylor (2019) that Timothy Ferris is at least a little bit self aware when he mentions that most great music is going to be excluded. I was curious who made up the committee chaired by Carl Sagan in determining the playlist of the Golden Record and by all accounts they are a team of white people (Ferris, 2017). It seems fairly arrogant, narrow minded, and colonialist that a team only made up of white people would be able to accurately and justly curate a playlist that represents humanity. I wonder about the cultural permissions of some of the songs that were included. Just because the Golden Record Committee secured song rights doesn’t necessarily mean they went through the process of ethically getting the correct permissions from the knowledge keepers of the songs. Ironically, in this task, I am just another white person curating the list down even smaller, I am no more qualified to say what should go on an intergalactic best of album. With that in mind, I did try and stay true the original goals of the record which is to be inclusive as possible and to make a good record (Taylor, 2019). 

With respect to inclusivity, I tried to pick songs that 

• represented a variety of sounds and instruments
• represented disparate geographic areas
• balanced out a range of male and female vocals 

With respect to making a good record, well that’s subjective. When I was choosing between songs in similar geographic regions, I picked the song I liked best. Additionally, I tended to favour songs with vocals rather than instrumental alone. I justify this choice since the album is human centric, why not reflect the actual sounds humans make. 

Here is my list of ten songs from the Golden Record with brief rationale:

1. Alima Song – This song from the Democratic Republic of Congo is beautiful to listen to and is the song that I chose from Africa. I favoured it because it also met the other criteria that I set for this task which was having a track list with more vocals and a track list that adds parity when it came to representation of gender. Although, this is a song that makes me question whether the culture who owns this song was consulted about its use. 
2. Jonny B. Goode by Chuck Berry was the first song I knew I was going to pick because it has the most personal relevance to me. My partner and I have been having an ongoing debate about collective cultural decay and whether or not we think the Beatles will be relevant in 50-100 years from now. From our research, including an email to beloved Canadian music writer and broadcaster Alan Cross, it seems that if any rock and roller is to be remembered it will be Chuck Berry, and specifically the song Johnny B. Goode. According to another music writer, Chuck Klosterman, the only memory of rock and roll at all will be of Chuck Berry. Seems like if we want to send any message to the cosmos about rock and roll, Chuck Berry is a good messenger.
3. Sokaku-Reibu (Depicting the cranes in their nest) – I like the symbolism of this song with respect to the Voyager.  According to this piece the music uses a technique called Koro-koro which is meant to imitate the flapping of wings of cranes and the song itself represents the raising of young cranes who eventually leave the nest. It seems as though this is meant as a metaphor for the two Voyager spacecrafts, from the birthing of the idea, to building them, to actually launching them into space (hence leaving the nest). From what I understand, cranes are also a symbol of a long and happy life, which seems fitting for the Voyager spacecrafts. I also chose this song for inclusion because of the unique instrumentation and because this song is from East Asia. 
4. Tchrakulo – Unlike some of the other regions of Earth (e.g. Europe), the Middle East was not entirely represented. The songs on the original track listing from Georgia and Azerbaijan come the closest to being from the Middle East. I ended up choosing Tchrakulo because it features humans singing. 
5. Melancholy Blues by Louis Armstrong and His Hot Seven was chosen because it met the criteria for including songs that had unique instruments on it, and this was the only track of the original set of tracks that featured brass instruments (with maybe the exception of the Stravinsky track). 
6. Symphony number five in C minor –  At least in the western hemisphere, Beethoven’s fifth symphony, first movement is still enduring and recognizable to this day, in other words it still has cultural weight. In fact, this song inspired a disco song in the 70s and was also used in Disney’s fantasia.  In addition, I chose Beethoven because I like the fact that Chuck Berry happens to have a song called Roll Over Beethoven which is a song about Rock and Roll replacing classical music. I like the juxtaposition of Chuck Berry and Beethoven on this track list. Also in choosing Beethoven I’ve chosen to eliminate Mozart as they were contemporaries and I think one classical artist is enough. Lastly this serves as my song choice from Europe. 
7. Navajo Night Chant –  Like the Alima song, this song makes me wonder if anyone from the Navajo Nation was consulted in its inclusion on the record. From a cursory search on the internet, it seems that this is quite a sacred song. I wonder if perhaps this song is only meant for those who are of Navajo heritage. Nevertheless, I chose this song as my North American pick. 
8. Solomon islands pan pipes – Of the three songs from the South Pacific I liked this one best. It also has the unique sounds of the panpipes, thus meeting the criteria for diverse instruments. Additionally, I did not choose the Australian Morning Star/Mokoi song because according to this article from the Atlantic  it might not mean what the original creators of the Golden Record thought it meant. In fact it might have a darker message than what was intended when it was sent to space.
9. Wedding song – It’s such a tragedy that the young women singing the song has never been identified or credited for the recording. It’s at least a beautiful sounding song and I am curious about the translation, I suspect it’s devastating. Additionally, I chose this as my song from South America and because it features female vocals. 
10. Bhairavi: Jaat Kahan Ho – This article translates the lyrics “Jaat Kahan Ho akeli, gori” as “where are you going alone, girl” and this seems metaphorically fitting for a spaceship alone in space. This song fits a lot of the criteria I set for myself when narrowing down the original track list and yet I hesitated to include this song because of something about the recording artist that I read here by Vikram Sampath: “Being an orthodox musician, Kesarbai was suspicious of the recording medium. She considered it a compromise on the art itself given the limited time available on a record. Her stand was in contrast to the other women musicians who preceded her and easily took to recording. She would often say that her music was not meant for someone sitting in a tea stall and listening to it casually while having a chat.”  Given her stance on recorded music, I wonder if Kesarbai Kerkar would have approved it being launched into space. 

Like I briefly mentioned, my partner and I have been having an ongoing debate about music’s relevancy and the collective memory. We wonder who will be culturally relevant in the future. If I made a Golden Record of music of the aughts, would anyone care in 50 years? What about 100 years? This task of curating the present for the future really elucidates Abby Smith Rumsey’s commentary on the difficulty of determining what data is valuable and needs to be preserved for the future (Brown University, 2017). In her talk at Brown University (2017), at the 38:20 minute mark Smith Rumsey states “what has long term value? We actually don’t know the value of anything until way in the future because its actual meaning is determined by events and contexts we don’t know about.” Ultimately, I do think the concept of the Golden Record is very cool, even if it ends up being kind of meaningless.

References

Brown University (2017, July 11). Abbey Smith Rumsey: Digital memory: What can we afford to lose [Video]. YouTube. https://www.youtube.com/watch?v=FBrahqg9ZMc.

Ferris, T. (2017, August 20). How the voyager golden record was made. The New Yorker. https://www.newyorker.com/tech/annals-of-technology/voyager-golden-record-40th-anniversary-timothy-ferris

Taylor, D. (Host). (2019, April 22). Voyager Golden Record (No. 65) [Audio podcast episode]. In Twenty Thousdand Hertz. Defacto Sounds. https://www.20k.org/episodes/voyagergoldenrecord

This task is a redesign of the original “what’s in your bag task” and when I think about the purpose of the original task I think about it from the perspective of identity texts. What do the things we carry say about who we are and our identity? I had discussed the concept of “mom pockets,” the game I play with friends about the things we carry in our pockets that are specific to our roles as parents. I have realized that much of my identity is in relationship to others and this cannot be fully conveyed through a written discussion of items. Dobson and Willinsky (2009) make the point that writing is formal and monologic, whereas speech is informal, interpersonal, and dialogic. Likewise, The New London Group (1996), draws our attention to the idea that different modes of communication draw forth different languages. I decided for this redesign that I would play with my understanding of modes of communication, identity, and texts by a) using social media for “mom pockets” to summon a check on my common identity with other moms and b) by recording a conversation with one of my closest friends about “mom pockets,” and identity.  To me, it’s a much richer discussion to talk about the things we carry as relational items and as texts situated in a shared identity. The interesting story that the ‘texts’ that I carry tell is what The New London Group (1996) might describe as the representation of a shared cultural context. 

When you listen to the recorded audio conversation of my friend and myself, not only do you hear us speak to a shared common identity, but you literally hear our shared common identity in our conversation pattern, our conventions of speech, and the language we use to make meaning. A theme that has come up in multiple readings (Dobson and Willinsky (2009); Kress 2004) is the loss of immediacy in written work. The audio recording instead affords an immediacy that my original written assignment lacks, and we can hear the give and take of a shared common identity.  If we consider the claim that Dobson and Willinsky (2009) make about writing (formal, monologic) vs. speech (informal, dialogic), we can see how they framed them in opposition. In this sense my first version of the task and this version of the task may be considered opposite modalities. I’ve also included a social media component which can be framed as a cross modality of both written and speech, in that it is written, but aims to mimic the informal nature of face-to-face communication. 

This audio clip represents the first half my discussion with my friend about “mom pockets.” The discussion includes the items in our pockets, our identities as parents, whether or not dad pockets exist, and a little discourse on the way woman, specifically, are socialized to parent. I’ve excluded the second half of our discussion as we digress into pandemic parenting.

Below is an image gallery of screenshots of how I use social media and “mom pockets” to summon a common identity

(Future Deirdre here: I saw that I could pull up very old stories in Instagram through the highlights option, so I’m able to include past, pre-pandemic “mom pockets,” including the infamous avocado pocket mentioned in my audio recording.)

References

Dobson, T., & Willinsky, J. (2009). Digital Literacy. In D. Olson & N. Torrance (Eds.), The Cambridge Handbook of Literacy (Cambridge Handbooks in Psychology, pp. 286-312). Cambridge: Cambridge University Press.

Kress, G. (2005). Gains and losses: New forms of texts, knowledge, and Learning. Computers and Composition 22(1), 5-22. https://doi.org/10.1016/j.compcom.2004.12.004

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review 66(1), 60-92.

Given the class’s ongoing discussions of the affordances of hypertext and the goal of creating a web of interconnection between the students of ETEC 540, a few of us virtually got together to play a game of Telestrations. For those unfamiliar with the game, it’s a mashup of pictionary and broken telephone where players take turns drawing and guessing phrases, objects, or actions.  Telestrations was a great opportunity to play with text, image, and hyperlinks.

You can follow the hyperlinks from our respective blogs to see how the game unfolded. It works best if you follow the hyperlinks linearly, but nothing is stopping you from jumping around.

There were six of us who played.  I started the game off with the drawing pictured below. Can you decode what phrase/object/action the image is trying to illustrate? Before continuing, feel free to put your guess in the comments, otherwise click on the image to see Sandra’s interpretation of my drawing. Click here if you want to see the original phrase that prompted the drawing.

If you’d rather just jump around from blog to blog follow the links to see:

• Sandra’s guess
• Kirsten’s drawing based on Sandra’s guess
• Nathan’s guess based on Kirsten’s drawing
• Ying’s drawing based on Nathan’s guess
• Selina’s final guess based on Ying’s drawing

It’s interesting to think about the ways in which computer-mediated communication and hypertext support a game like this and what is lost when we remove the immediacy of playing this game in person.

Emoji representation of a popular film. Numbering has been used for referencing.

I just recently had to watch the move that I’ve depicted in emoji for another class, so it was still fairly fresh in my mind. Beginning with the title, I chose emoji that best reflected the more salient plot points in the movie. The emoji used to represent the title for instance are, I think, the most iconic or memorable parts of the film. In telling the plot through emoji, I focused on individual scenes represented on separate lines.

Interestingly, Kress (2005) discusses how different modes of representation each have their own qualities that define their use. For example, in the written mode of representation the sequence of words is very important for meaning making and readers are dependent on the specific order an author has laid out. However, with respect to images, the elements chosen for representation are presented simultaneously (Kress, 2005). What is to be said of the affordances of emoji? In my emoji story, I’ve taken advantage of the affordances of the written word, in that the emoji need to be viewed sequentially, at least line by line. However, this isn’t as clear cut if we’re to just look at each singular line of my story. For instance, I need the reader to view the second line of my story sequentially, however for the ninth line I need the reader to take in the images simultaneously. What about the title? It doesn’t require sequential reading, nor does it really require viewing the images simultaneously. For the most part I could have put the emoji in any order, however I do need the reader to group some of the emoji together as one item; the red circle, the pill, and the blue circle need to be tethered together as one image rather than three separate images.

When trying to depict narrative with emoji, there is also a clash between the conventional sequencing of words with respect to objects and actions and the fixed directionality of specific emoji. Take a look at line ten in my emoji story and consider the meaning that you make. Who is shooting whom in that sentence? The toy gun emoji is in a fixed position. Contextually, we know the direction that a gun fires and those on the barrel end of a gun are on the receiving end of a bullet. However, as Kress (2005) notes, the first position in a written sequence has specific meaning. It could mean that the person placed in the first position is causing or responsible for the action. With respect to this emoji story, is the first person in line ten being shot at or are they the one doing the shooting? In other words, does the direction of the image (the toy gun) take precedence for meaning making, or does the linear sequence afforded to sentence structure take precedence?

In Bolter’s (2001) discussion of picture writing, he makes an important point about its lack of narrative structure; “the picture elements extend over a broad range of verbal meanings: each element means too much rather than too little” (p. 59). Conversely, Kress (2005) describes words as being vague and empty of meaning without a reader to interpret them. If both text and pictorial depictions are vague, then perhaps the meaning making magic happens when they are combined. Personally, I can relate to using emoji to enhance my written word in casual conversations through text. Because I have been told I have a blunt style of written communication (I just don’t see the need for a lot of formality and exclamation marks), my tone is often misinterpreted. If I throw in an emoji or two I can quickly convey that I mean no trouble. Likewise, I have been known to clarify the tone of a text by recording myself reading the text with the intended tone. Consider this very real text exchange between myself and my younger sister (for context, my sisters and I were trying to virtually meet up, but having scheduling conflicts). 

Me: You live your life. Join us whenever the scheduling works out for you! 

Sister: you sound sarcastic AF

In her defense, I can see how my message changes significantly with tone. My intended tone was meant of support and flexibility. I ended up sending a voice recording in order to speak the text as intended. In text messaging we augment our writing with emoji, gifs, voice recordings, and the tapback feature on individual text bubbles on iPhones. I’m reminded of Kress (2005, p.17):

As one effect of the social and the representational changes, practices of writing and reading have changed and are changing. In a multimodal text, writing may be central, or it may not; on screens writing may not feature in multimodal texts that use sound-effect and the soundtrack of a musical score, use speech, moving and still images of various kinds. Reading has to be rethought given that the commonsense of what reading is was developed in the era of the unquestioned dominance of writing, in constellation with the unquestioned dominance of the medium of the book.

Makes me wonder about the representational changes of reading and writing in the dominance of the medium of the smartphone. 

References

Bolter, Jay David. (2001). The breakout of the visual. In Writing space: computers, hypertext, and the remediation of print. Routledge. (pp.47-76).

Kress, G. (2005). Gains and losses: New forms of texts, knowledge, and Learning. Computers and Composition 22(1), 5-22. https://doi.org/10.1016/j.compcom.2004.12.004

It took more hours than I care to reveal, but I’ve created my very first Twine! I don’t mind that it took me a long time, it was pleasantly frustrating and the process was as enjoyable as the product. In this game called Get out the door you’ve slept through your alarm and now you’re running late. Your job is to get through the game before 8 AM. There are a few paths to victory and several dead ends. I hope you enjoy the game and I’d be happy to hear what you think.

Get out the door (2)

I’ve never  used Twine before and there was indeed a learning curve. I think that one needs to have a decent level of frustration tolerance before embarking on making a product on Twine. I would imagine that the more I used the program the more efficient I would become at using it. However, for me personally, part of the joy of making the game was the problem solving aspect, not just in coming up with the idea, but in actually employing the program. Kafai (2006) discusses the importance and value not in playing games for learning, but in making games for learning. I can see how Twine would certainly support the idea of ‘making’ as learning. For instance, my peer, Ying used Twine to tell an informative procedural story. Having students use Twine to demonstrate concepts would be an excellent way to make games for learning. Twine could also help students illustrate and make explicit their thinking and understanding of topics, especially maybe for students that are challenged by showing their thoughts linearly. As Bush (1945) notes the mind works by association and connects ideas through a web of trails, so why not capitalize on that associative experience by letting our students demonstrate their thinking that way too. Again, Ying has a really great discussion of this on her page.

My process for creating my game on Twine evolved organically and was not premeditated. What ended up happening was that anytime a fork in the story was created, I would continue fully down one path before returning to the fork. In that sense, there was a linear process included in making a game with parallel narratives.  Bolter (2001) writes that “all writers have had the experience of being overwhelmed with ideas as they write” (p.32) and discusses the idea that before the digital age and before printing, that there was a sense that writers were overwhelmed from within. I think that being able to use Twine to write parallel stories could ease that sense of being overwhelmed. It’s almost as every fork that was created served as a bookmark of an idea to come back to. This got me thinking about all the ways I bookmark ideas in my own life.  One habit I have is having multiple tabs open on my computer of similar related ideas that I can come back to.  In the last few months I have been using concepts maps. I have multiple notes in my notes apps to come back to. And sometimes I just store Ideas in my head, hoping that I’ll remember to come back to them.  I appreciate hypertext in this sense, because I can use it to follow streams of consciousness (I assume we can all relate to going down a Wikipedia rabbit hole).  I can see how something like hypertext would  help to solve the sense of both being overwhelmed from ideas from within and overwhelmed from all the information available for us to assimilate from the outside in (Bolter, 2001).

After reading about Nelson’s (1999) Xanadu, I was surprised to see how much my Twine resembled his preliminary ideas on parallel documents. Nelson discusses the idea of parallelism between documents and the need to be able to show, visually, the connections between them in order to compare them. I had never heard of Xanadu before, and as far as I can tell it has never come to fruition, but certainly I can understand the perspective, like that of Bush (1945), of wanting to be able to find a better system for organizing all the information that is available in the world. I can also understand the desire to find a system that breaks free of the limitations of the material world.

The back end of a game created in Twine may resemble the preliminary drawings of Theodore Nelson’s (1999) Xanadu.

I’m curious, how do you keep track of your thoughts, do you feel like you think more linearly or more associatively? How do you ‘bookmark’ ideas in your day to day life?

References

Bolter, Jay David. (2001). Hypertext and the remediation of print. In Writing space: computers, hypertext, and the remediation of print. Routledge. (pp.27-46)

Bush, V. (1945). As we may think. The Atlantic Monthly, 176(1), 101-108. https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/

Kafai, Y. (2006). Playing and making games for learning: Instructionist and constructionist perspectives. Games and Culture,1(1), 36-40.

Nelson, T. (1999). Xanalogical structure, needed now more than ever: Parallel documents, deep links to content, deep versioning and deep re-use. ACM Computing Surveys, 31 (4).

A picture I took of my writing while channeling my best inner millennial influencer

As an older millennial, computers were part of my school experience, but mostly we did our work by hand. I remember having a very prominent writer’s bump on my right hand that I was self conscious about. Over time, without me even noticing, my writer’s bump eventually disappeared ― a signal that indicated a shift from primarily manual writing to typing. I found the exercise of manually writing 500 words physically straining. Where my writer’s bump used to be was a purple, fleshy indent that became very tender.  I can’t remember the last time I manually wrote 500 words in one sitting.  

Compared to my typing, this was by far a more time consuming activity. When I made mistakes in my writing I either tried to correct the mistake, by writing directly on top of it (e.g. changing a lowercase ‘t’ to an uppercase ‘T’), or I crossed out the word with a line and continued writing. In the case of a missing word, I wrote the word overtop with a little arrow indicating where the word should be inserted. I used a pencil to complete this task and since I didn’t use an eraser, I believe I would have edited my work the same way with a pen.

It’s hard to say which form of writing, manual or typing, I prefer and I am reminded of something host Brad Harris said in an episode of The history of the modern world, “the value of a printed book is its content, but the value of a handwritten book was mostly the object itself” (Harris, 2018, 1:58).  There is a charm to writing by hand and there is something to be said for the aesthetic appeal of manual writing. Doing this exercise has made me nostalgic for my late grandmother’s penmanship, or the notes my mum used to write me in my packed lunch. I enjoy manual writing when I’m doing something stylistic (e.g. crayligraphy or brush lettering) or personal, like writing a letter to a friend. I also write manually when I know it will take less time to write something down than to open a computer (e.g. grocery list). On the other hand, I prefer typing for assignments, lesson planning, daily correspondence, and anything that requires professional or formal writing. In my opinion the benefits to typing are speed, uniformity, ease of editing, and ease of sharing documents.  

References

Harris, B. (Host). (2018, February 5th). The printed book: Opening the floodgates of knowledge [Audio podcast episode]. In How it began: A history of the modern world. https://howitbegan.com/episodes/the-printed-book/.

Below is a five minute unscripted story of my family first processing the COVID -19 pandemic as captured by speech-to-text technology in the notes application of my iPhone. I’ve gone ahead and annotated one of the texts in order to classify the ways in which the text deviates from written conventions of English (scroll to the end to compare to the second text). I’ve sorted the differences into two categories: 1) Errors of the speech-to-text technology (represented in green) and 2) differences in spontaneous speech to written text (represented in purple). As an additional exercise I attempted to retell the same story, but using the speech-to-text technology on my Microsoft Surface. Telling the story twice was interesting for two reasons: 1) It was interesting to see if one speech-to-text application was more accurate than the other, and 2) It was interesting to see the way my story changed;  the details I did or didn’t include on the second retelling, even though I told the story back to back.

Unscripted oral story captured by iPhone speech-to-text technology

Both of the text-to-speech technologies differed similarly from the writing conventions of written English. The technologies didn’t make the same exact errors, but made the same types of errors. Overall, the text reads like a stream of consciousness and I think this is both due to how spontaneous speech occurs and to errors in speech-to-text technology, for that reason I did not ascribe lack of punctuation or lack of written structure specifically to either category. It is difficult to determine what the most common errors are as I don’t have an exact copy of my spontaneous story and therefore do not know precisely the nature of all the errors. Because there is a 24 hour lag from when I used the technology to when I began to review the technology,  I can not recall what may be missing from my story or what the speech-to-text technology has added. The most salient errors appear to be recognition errors. Interestingly, as Gnanadesikan (2009) notes, writing was invented to solve the problems of memory. Had my story been scripted and written down, I’d have an exact copy of what I said to compare the speech-to-text text, instead I’m relying on my own memory which is susceptible to error (Shaw, 2016).

I’ve never used speech-to-text technology before and have never spent time considering it. What has become very apparent to me is how poor it is at a means of a stand alone form of communication. In absence of an accompanying voice recording, we lose sense of tone, pacing, volume, and other conventions of storytelling. In absence of video recording we further lose gesture and facial expressions. In absence of a written script, we lose punctuation and flow. The rich details of communication are lost if we rely only on the raw transcriptions of speech-to-text technology.

If we’re just using the speech-to-text technology to analyze orality, which I think is the point of this task, the speech-to-text text reveals to us the spontaneous nature of orality which, in my case, would differ quite significantly from my written text. Specifically my written text would omit instances of repeated words, self corrections, filler words, and informal pronunciations/spellings. Likewise, I would have included much more detail in my written work. As Boroditsky (SAR, 2017) notes, when we speak we are only conveying a small portion of information and hoping that the listener fills in the gaps. I would say that this is true for my spontaneous oral story. I’ve only conveyed a small portion of information in my oral story, whereas I would have been more descriptive in my writing.  In chapter one of Orality and Literacy: The technologizing of the word, Ong (2002) explains that as the technology of writing progressed it went from transcription of oral speeches to being produced specifically as written text. In other words, it went from being a method to record an instance of oral utterance to being used strictly to produce an idea in written form. The act of using language to produce a piece of written text changes the way we use language compared to using it to tell an oral story. To return to the work of Gnanadesikan (2009), she states, writing is a time machine, and to that end  I can copy and paste my writing exactly ad infinitum. However, having used the speech-to-text technology twice by retelling the same story, I can see that oral transmission of information differs from one telling to the next. Perhaps this quality of orality could be positioned as a problem that needs to be solved, yet this might also be a benefit of orality. Iseke and Moore (2011) discuss the transformations of oral stories to video, specifically in the context of Indigenous elders and traditional storytelling. Though their discussion focuses on video recordings, I think it can be extended to the idea of transforming a story from oral tradition to written form. According to Iseke and Moore (2011), when we restrict a story to video or text we lose out on the ability to modulate the story for the needs of our audience. Children, for example, may require a different version of a story compared to adults. Recording a story necessitates a more generic version of a story and eliminates nuance and complexity that comes from being able to make adjustments for a variety of audiences (Iseke and Moore, 2011). Indigenous storytellers have developed the skills needed to determine what an audience knows and to help decide context for their stories, this is quite distinct from what happens when we freeze a story in time by recording it.

Orality, compared to the written word, differs in quite numerous ways, from the language we adopt, to the formality of the story we tell, to the details we choose to share and everything in between. This is only a small discussion of those differences and certainly there is more to be said.  I’d be interested in knowing about the oral traditions in other families, the stories we pass down from generation to generation, and the needs we have to preserve them. As I write this in the bathroom while my 5 year old takes a bath, she is asking me to tell the story of when she was born. I’ve told this story to her before and she loves listening to it. I’ve never written it down.

Unscripted oral story captured by Microsoft Surface speech-to-text technology

References

Gnanadesikan, A. E. (2011).The first IT revolution. In The writing revolution: Cuneiform to the internet (Vol. 25, pp.1-12). John Wiley & Sons.

Iseke, J., & Moore, S. (2011). Community-based indigenous digital storytelling with elders and youth. American Indian Culture and Research Journal, 35(4), 19-38. doi:10.17953/aicr.35.4.4588445552858866

Ong, W. J. (2002). The orality of language. In Orality and literacy : The technologizing of the word (pp. 5-15). Routledge.

SAR School for Advanced Research. (2017, June 7). Lera Boroditsky, How the languages we speak shape the ways we think [Video]. Youtube. https://www.youtube.com/watch?v=iGuuHwbuQOg&t=516s

Shaw, J. 2016, August 8. What experts wish you knew about false memories. Scientific American. https://blogs.scientificamerican.com/mind-guest-blog/what-experts-wish-you-knew-about-false-memories/