Play on itch.io (may take some time to load; may not work properly for all browsers and devices): https://etec565sdruids.itch.io/mode-bending-whats-in-my-bag

OR

Download to run on your computer: click here to download

 

(please note that you will likely need to scroll left-right and up-down on your device to access all parts of this multimodal work)

 

To begin this project, I started by thinking about the original purpose of the “What’s In My Bag” task: presenting the private contents of my bag in order to “implicitly prompt the viewer to begin to construct a narrative about” me. The original task was entirely visual, with the contents of the photograph taking on the roles of texts to be interpreted by the viewer.

In shifting modes, I wanted to see if I could re-imagine the ideas behind the intent of the original project without relying on the core visual element (and without re-introducing written language text… trying to convey meaning without any written language was a fun challenge from the previous task that I wanted to explore again here). I decided that I would represent the original images through emoji. To lend further context to each object/representation, and to redesign the original “glimpse of my private life” aspect, I created audio clips meant to represent a combination of audio “snapshots” of my day and internal monologues made audible. To tie these audio clips to the emoji that represented the objects in my bag, I created an interface using Twine that the user can freely interact with to freely explore each object/sound combination in any order.

To organize these connections between digital pictographic symbols and audio clips, I introduced another Available Design that also allowed me to lend additional weight to the “glimpse of my private life” aspect of the design: the literacy or “grammar” of a cartesian plane-style timeline (New London Group, 1996). With the x-axis labelled by time (using emoji that call upon the additional literacy of reading the face of an analogue clock), I was able to represent and chart each item’s use or uses to specific times in the day. In cases where an item is used more than once, this created opportunities for me to connect multiple audio clips for the different instances of each item’s use, creating additional context for the user as they use the representation of these items construct their understanding of who I am.

The presentation of this timeline of emoji in a html file produced in twine invites the user to click on linked elements in the hypertext interface to interact with the “text” in an order of their own choosing. In this way, the use of Twine and html helps to “create patterns of meaning that are more or less predictable in their contexts” (p.22), as the medium inherently suggests the active interactivity of the user that is required to “read” the text, rather than a more passive form of consumption.

The biggest challenge I encountered when working on this task was ensuring that the organization of emoji stayed spatially organized relative to each other so as to not disrupt the precision necessary for the grid-coordinates system, which is important for that element to maintain its meaning. Originally I used pairs of emojis (and colour coding) to construct concepts such as “wallet” (credit card + folder) that do not have an obvious representation in the emoji character set, but I soon found that the thin columns that the grid is made from would often force the second character  “wrap” down to the next line of text, displacing every other symbol in the column below.

References

The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. (Links to an external site.) Harvard Educational Review 66(1), 60-92.