A Multimodal Approach: Gestural + Audio
For this task, I decided to do a mode-bending that included two new modes not used originally for task 1: gestural representation and audio representation. Originally I was going to do only a gestural representation but then I thought that including an audio representation would allow me to give more clues to the viewer, as I am presenting the objects without giving away their identity through words (although there is a link to reveal that information).
Using gestures, I am expressing how each object is used. Using sound, I am expressing what sounds can be produced with the objects or activities I do with the objects. It is interesting to contemplate how, compared to the original task, there is no visual nor textual representation of the objects, and how the immediacy of identifying them is reduced. This made me think about how visual and textual representations are usually the most precise modes to describe objects. Seeing an object and its name makes possible instant recognition. I believe it mirrors how the mind works, as it seems in my experience that, in thinking, a name can’t be separated from form (try to imagine a nameless form!). However, what I also learned from this mode-bending task is theat audio (not as speaking, but sound effects) and gestures are very efficient to illustrate actions or dynamic content. This made me think about how when we think about verbs, the images produced in the mind are dynamic and, in that sense, more like a gesture or a sound effect.
Hence, this redesign process was one of going from static content to dynamic content; from objects to activities; from nouns to verbs. However, if I look at it more critically, it seems like the act of representation is in itself a dynamic process of transformation. Even in the case of the images and the words, there is a initial process in which the objects (which is a perceptual experience) are converted into a static form (pictures and words). This help me to understand Kalantzis and Cope’s (2009) observation that “all forms of representation, including language, should be regarded as dynamic processes of transformation rather than processes of reproduction” (p.175). Another important idea expressed by Kalantzis and Cope’s (2009) and in ETEC 540 is that experiences are multimodal. For example, the gestural representation was clearly visual as well, although gestures could also be expressed only through sound or words. But could we have gesture separated from a visual, auditory, or linguistic experience?
Another interesting observation that came from this task is that my gestural and visual representations are biased. To read these visual and gestural texts the observer must have a cultural background sufficiently close to mine. An individual from 2000 years ago wouldn’t be able to understand most (if not all) of the representations. The gestures and sounds are informed by cultural objects that belong to a particular culture and time. However, it is interesting to think about how some of the emotions I display in the videos (for example enjoyment or peace) have a transcendental quality that could be understood by any human. This made me think about Dobson & Wilinsky’s (2009) assertion that “literacy should be viewed, rather, as a set of complex characteristics and processes that influence and are influenced by social context and personal circumstance” (p.15). From my analysis of how the objects biased the representations, I can see the truth in this statement. However, from the analysis of the transcendence of some texts (for example emotions), I believe it could be argued that maybe there is some literacy that is inherent in human beings, regardless of context or culture.
Object 1
Reveal Object 1
__________________________________________
Object 2
Reveal Object 2
__________________________________________
Object 3
Reveal Object 3
__________________________________________
Object 4
Reveal Object 4
__________________________________________
Object 5
Reveal Object 5
__________________________________________
Object 6
Reveal Object 6
__________________________________________
Object 7
Reveal Object 7
References
Dobson, T. M., & Willinsky, J. (2009). Digital literacy. (pp. 286-312). Cambridge University Press.
Cope, B. & Kalantzis, M. (2009). Multiliteracies: New literacies, new learning. Pedagogies: An International Journal, 4:3, 164-195, DOI:Links to an external site.
10.1080/15544800903076044Links to an external site.
Hi Eduardo,
I really enjoyed the creativity you brought to this task and it was a fun interpretation to turn it into a game. Your observation on cultural bias in terms of representation and interpretation is also insightful as the use of gestures could mean something quite different from one culture to another and alter the intended communication. I also wondered if the emotions that you were portraying could be a form of bias in the sense that they are framing how the audience knows you feel about the objects.
With gestures, there is the purposeful exaggeration of movement to draw attention to the object being referenced. In this way, it feels less natural than viewing a true interaction with the subject and object. In the absence of the object, the viewer’s eyes are more aware of the gestures that may be less important in an authentic interaction.
The addition of sound certainly aided in recognizing some of the objects but I would argue not all. One observation I made was how the sounds become amplified in this context and are far more noticeable than in an authentic situation. As an example, the chewing sounds become dominant in the banana eating scenario to the point of distraction but I wonder if this is an individual response or one shared by many? I could certainly understand the object in question was a banana through gestures, but I would not have been able to draw this conclusion with sound alone.
Thank you,
Chris
Hello Chris,
Thanks for your kind comments. Yes, this was useful to reflect on how all representations are subject to bias. Perhaps understanding this is essential for us to engage more openly with others, understanding that our ways are largely conditioned and not the ultimate truth. It seems to me that emotions are universal, however, I do see your point – the emotional relationship to the objects I relate to is definitely biased by my own view of what they represent to me. Great observation.
Good observation about how, in some cases, the sound didn’t reveal as clearly the object as the gestures. This was particularly challenging with the hat and the banana eating. I think the conclusion we can make from this is that some objects have more presence in different modes of representation.
Best,
Eduardo