Communication technology: Graphic Notation (music)
Medium: Website presentation

Visit:
https://edurebagliati.wixsite.com/graphicnotation

1. Utopian Speculative Future of Music Production

By the year 2051 technological advancement has taken music production to levels no one would have imagined in the 20th and early 21st centuries. It was all possible through the development of holographic computers, which are extremely powerful and interconnected. A worldwide agreement was made to guarantee that every human has access to a holographic computer, which is not dependent on physical material. The computer is a highly intelligent system that operates in conjunction with the user’s mind.

Digital Audio Workstations (DAW) have become very powerful, allowing music producers to create unprecedented materials. Its features include:

• Virtual session musicians. A function that enables users to have their musical ideas performed by expert real-life musicians whose performance abilities have been algorithmically translated into the program. This has allowed anyone to be able to create pieces of high-level performance, as well as creating an interconnected community in which those who have been trained to perform can offer their services to anyone around the world.

• AI sound processing. A function that helps users achieve perfect sound processing for mixing and audio alteration. As tracks are created an AI sound assistant ensures perfect balances in levels, equalization, compression, stereo imaging, reverbs, amongst others. These balances are defined through an interface in which users determine what genre of music they are working on. The program was developed and is constantly updated by sound engineers all around the world.

• The infinite library. All existing instruments have been virtualized and can be assigned to the virtual session musicians.

• Music knowledge download. This function allows users to download theoretical knowledge into their minds, giving them the instant capacity to compose, arrange, and create in any genre of music. This was possible through the creation of mental directories built on the knowledge of expert musicians from all around the world. When humans discovered that all individual minds are part of a collective human mind, and built a download system, the dynamics of learning were revolutionized.

Hence, music production has been released from many limitations and has become an activity that is more focused on creativity and innovation. Great music productions are created through the producers’ ability to intelligently combine the different functions of modern DAWs. Similarly, given the ubiquity of holographic computers, the virtual musicians function, and the music knowledge function, everybody has the capacity to produce music and express their unique selves through this medium. The culture of music and music production around the world has become one of appreciation of ideas and equity in which technology empowers human capacity. 

2. Dystopian Speculative Future of Music Production

By the year 2051 technological advancement has taken music production to levels no one would have imagined in the 20th and early 21st centuries. The advancements in Artificial Intelligence have transformed the way in which Digital Audio Workstations (DAW) operate, allowing all functions to be automated with minimal human intervention.

After all human skill was decoded to be applied using DAW’s virtual instruments, performing musicians are in extinction. This in turn changed the dynamics of the music industry, as humans are no longer at the forefront of live music performance. Instead, holographic avatars that connect to the decoded human skills system have become the idols of the entertainment business. Idolatry of non-human figures has led to many psychological, sociological, and philosophical issues, making people more alienated and unable to have empathy.

Traditional music producers are also becoming extinct as the Artificial Intelligence system of DAW is able to produce much more rapidly and commercially efficient. Powerful corporations use big data to understand sociocultural behaviors of consumption and incorporate that knowledge in DAW functionality to create pieces of monumental sales and subliminally promote their products. Hence, the music production industry is dominated by corporations that are able to use the algorithmic functions of the highly advanced DAWs alongside consumption data. This has made music a means for advertising and sales instead of channel for self-expression and artistic freedom. Even those individuals who are looking to use the DAW for creative and artistic purposes have been neglected, as the new industry dynamics have created a new type of “listener” in modern cultures, who is dominated by impulses and drives which are perfectly understood and manipulated by the corporations.

The total virtualization of musical instruments in music production software disrupted the industry and culture of musical instruments. Almost all companies that produced instruments have become extinct, as they were not able to compete with virtual instruments, which can be loaded in all devices such as phones and holographic computers, and performed through AI assisted programming. The small group of those interested in acquiring an instrument have to pay $50000 for an electric guitar or $450000 for a grand piano. Even when an instrument is acquired, the fast-paced dynamics of daily modern life makes it almost impossible for an individual to learn the instrument. Those who want to make a living in music production or music have no choice than surrender to the system and only those who are to produce for marketing and mind-control purposes succeed.

I thought the topic of this week was fantastic and really helped me to understand how algorithms function, their potential and dangers. The articles by O’Neil (2016, 2017) and the podcast by McRaney (n.d.) were particularly enlightening to me.

Rather than doing one microblog post using predictive text assistance, I was interested in trying out different text predictors to see what results I would find. I did the microblog post using two websites (Inferkit (Case 1) and Deep AI (Case 2) and my cellphone (Case 3), which I did in Spanish, given that it is the language set in my phone.

I was particularly surprised by the text created using Deep AI (Case 2) as it had a political tone that felt the most different from the way I would express myself. The post contains a quote that critically mentions Donald Trump. Anyone who knows me would know that I rarely speak about politics or politicians – it’s simply not an interest of mine. However, if I was to post this on my social media platforms it could easily be inferred that I have an inclination for political discussions and that I have a specific political position about a politician. In McRaney’s (n.d.) podcast it is mentioned that algorithms can have unintended consequences and we should think carefully about how they can affect people. The post is not dangerous in this explorative educational context but I do imagine how algorithms could lead to expressions that contain political, sexist, racist, and ethical implications that have nothing to do with our ways of thinking and create tension in our lives, as well as affect people.

The text created using Inferkit (Case 1) was also surprising, particularly because this platform didn’t create text on a word-to-word basis, but rather in sentences. The result is a very well-articulated and philosophical idea about the meaning of education. This made me think about Dr.Vallor’s (Santa Clara University, 2018) definition of AI as machine-augmented cognition for humans, because the experience of creating text through Inferkit felt like a more potent way of creating ideas compared to the word-to-word approach. I understand that Dr.Vallor’s  (Santa Clara University, 2018) refers more directly to using machines to perform extremely complex tasks in seconds, but I think it also applies in this case. This makes me wonder about some ethical issues about honesty in educational contexts. I actually went beyond the 250-word limit to see what else came up and the final result contained more complex well-articulated ideas. It even included quotations. I thought about how students could use this tool to cheat in schools, which is not only an ethical issue but also an educational one, as it represents an omission of doing tasks that are imperative for the development of skills (for example critical thinking or abstraction) that are needed to be successful in our society.

The text created using the cellphone (Case 3) provided insight into different aspects not found in the other two cases. I observed that, in this case, the algorithms were influenced by texts I’ve previously created from using my phone on a daily basis. I had a more clear experience of what was mentioned in McRaney’s (n.d.) podcasts about how algorithms predict the future based on the past. Some words such as “vida” are used a lot when I am communicating through instant messaging and it was interesting to see how it was suggested many ways while doing this task. I appreciate how algorithms can predict the future based on the past and have found that it is helpful when producing text in instant messaging. I understand now that this is one of the functions of algorithms, as they are precisely created to predict outcomes, yet I also see how, in other contexts, wrong ideas from the past (which could be racist, unethical, or sexist) could affect decisions we make in the future and create problematic feedback loops in algorithmic structures. This is the danger for which transparency and care from designers are urgently needed.

I was also left with the impression that the Spanish case was not as sophisticated, which makes me wonder about the technological hierarchies that exist between cultures, as in other cases such as MOOCs production.

Case 1: Inferkit

Text: Education is not about proving yourself, but about connecting with others and learning from them. Education is not about making you feel that you’re ready to perform or compete. It’s about engaging, being active, and getting excited about learning.

Case 2 : Deep AI

Text: Education is not about politics, it is about understanding the political world,” said Tom Reza, an assistant professor at Georgetown University’s Division of Criminal Justice. “All I’m concerned about is if President Trump believes he can achieve this or not he should be impeached if it does not happen.”

Case 3 : Cellphone:

Text: La educación no se trata de un problema de aprendizaje que se puede cambiar en la historia de la humanidad. El problema es que no se puede entender que es una forma de vida que no es fácil porque se trata de una forma de vida que nos permite vivir en una sociedad que se puede cambiar.

References:

McRaney, D. (n.d.). Machine Bias (rebroadcast). In You Are Not so Smart. Retrieved from https://soundcloud.com/youarenotsosmart/140-machine-bias-rebroadcast

O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy (First edition). New York: Crown.

O’Neil, C. (2017, July 16). How can we stop algorithms telling lies? The Observer. Retrieved from https://www.theguardian.com/technology/2017/jul/16/how-can-we-stop-algorithms-telling-lies

Santa Clara University (2018, November 6) Lessons form the AI mirror Shanon Vallor [Video] Youtube.com https://www.youtube.com/watch?v=40UbpSoYN4k&ab_channel=SantaClaraUniversity

Exploring the GUI was a fantastic experience to see more clearly the ‘dark patterns’ designers employ to lead the manipulate our behavior in digital environments. From the moment I opened the GUI I noticed I was in a very deceiving environment which purpose was to maximize the time I spent on the platform and the number of clicks. This made me think about Harris’ (2017) TED Talk about the persuasive techniques that tech companies use to control our minds. In effect, Harris (2017) mentioned that “the best way to get people’s attention is to know how someone’s mind works”, an inference we could make about those who designed this GUI. For example, we often feel stress by time pressure, a condition that was created with the GIU’s timer. Looking at it more deeply, the timer is instigating a sense of losing something (bringing up thoughts like “I’m running out of it!”) which created a condition of fear. Under such conditions we can lose concentration and clarity, which makes our behavior more erratic, meaning more susceptible to clicking the wrong buttons – which are the buttons they really want us to click. Fear could also lead to a paralyzed state in which the time we spend on the environment is maximized.

Another dark pattern I was able to quickly recognize based on my experience as a web and graphic designer was the intentional misuse of color to deviate our attention from the steps to complete the game. Often, the colors used in words that allowed progression in the DIU were similar to the background color. Similarly, the words were often small, not underlined, and positioned away from the DIU’s focal points. In this case, rather than an understanding of how our mind works, I would argue that some dark patterns are built on an understanding of how our sensory perception works to manipulate attention. I worked in the advertising industry for a couple of years, doing audio and multimedia productions for marketing campaigns. In the same way advertisers understand how visual perception works and this enables them to create visual objects (commercials, short videos for social media, posters, flyers, etc) that manipulate behaviour, they also create auditory objects for the same purposes. Something that often surprised me while producing the audio for commercials was how announcers and voiceovers were always directed to express in overenthusiastic ways. Many times, while recording, directors and clients would say “go higher”, “raise it”, or “do it with a bigger smile” to a point in which, in my view, sounded unrealistic (seriously, how excited can you be about buying a Pepsi?). I think that when a dark pattern is taken to unrealistic dimensions its deceiving intentions are more clearly exposed.

The concern I am left with is about the subtle dimensions of deception, which are much more difficult to scan. The GUI’s tricks and intentions were very easy to identify, but I am sure in other digital environments billions of people use daily (Facebook, Instagram, Google, etc) deception is way subtler. This would be like being approached by a thief whose whole body language and appearance clearly reveal his identity, and being approached by another thief who is disguised as an unharmful friend. For this reason, I think it is essential that we are educated about media deception and develop the necessary literacy skills that allow us to see through the lies. As mentioned by Tufekci (2017) the persuasion architecture of digital environments is much more sophisticated and potentially dangerous. It is important to understand also that this is not just a matter of selling products or manipulating behavior, but it also has the capacity to manipulate belief systems which are the foundation of an individual’s behavior.

References

Harris, T. (2017). How a handful of tech companies control billions of minds every day. Retrieved from https://www.ted.com/talks/tristan_harris_the_manipulative_tricks_tech_companies_use_to_capture_your_attention?language=en

Tufekci, Z. (2017). We’re building a dystopia just to make people click on ads. https://www.ted.com/talks/zeynep_tufekci_we_re_building_a_dystopia_just_to_make_people_click_on_ads?language=en

In this linking assignment, I’m going to make connections and analyze differences between the reflections my classmate, Tanya Groetchen, and I did for Task 9 – Network Assignment.

I found that Tanya did a good job at organizing the data to read it more clearly. She provided screenshots of how she moved around the nodes, separating the music tracks from the curators. This was a good way of separating the two different kinds of data from the graph. The size of the nods in curators doesn’t represent any information, however, for the music pieces, it represents how many times these were selected. Although simple, I think this was a very skillful approach and made the reading of data much easier. I didn’t use this strategy and see now how it could have saved me some time in data interpretation.

This makes me think about how important data organization is and how agency is needed for such a task. This could easily have been done with artificial intelligence but it seems like that function is not featured in Palladio. It’s interesting to contemplate how computers are crucial for the assembly and reading of data. Without such agency, how we read and find data today would be very difficult. Search engines and data analyzers precisely organize massive content in ways that make our life easier. However, as we’ve learned from this task, the computerized assembly of data can only go so far. This data has no way of providing information about what were the implications behind the patterns in the curation, and this is where we have to be extremely careful because we can make wrong assumptions. Does the fact that the Fifth Symphony by Beethoven was the most popular track mean that everyone likes classical music? Or does the fact that Johnny B.Goode was also among the most popular tracks means that most of us would enjoy ourselves in a rock n’ roll concert? Or maybe it shows that it is very likely that we’ve all seen Back to Future? Like this, we can ask lots of questions without a clear conclusion. I don’t think this is something negative – quite the contrary. Data can stimulate our minds to build assumptions through which we can question beliefs and expose bias. In this example that might not be of huge significance, but in other contexts, it could be a great way of learning about assumptions we make on race, gender, and culture. This exposure can help us to remove judgments, discrimination, and unethical practices.

Both Tanya and I explored the issue of the superficiality of the data, concluding that any reasoning on the motives behind the curation remains at a speculative level. To understand the deeper dimensions of why individuals choose certain things more data is required. I think that human behavior is so complex that to understand the mechanics of choice profoundly, we would probably need more data of a qualitative nature. I find it fascinating how, as we go deeper into the motives behind choices, at a certain point, there is most likely a shift from the individual to the collective, meaning that the thought which produced a choice might be only an impersonal thought that comes from a cultural heritage. In that sense, data can also provide us with insightful information about culture and groups of individuals. I appreciate doing this data analysis task, as it has helped me to understand the necessity of being careful while interpreting data, and also the capacity data has to storage individual and cultural patterns.

I am left with an intriguing question: is our personal makeup a complex set of data? Maybe this is why when we lack profundity in our ways of viewing people, we only see superficial data and make assumptions and wrong judgments. If we were to de-compose the data of an individual to the point that it is seen as a cultural construction, maybe we could see the essential ‘blank canvas’ of an individual (essence) while understanding that we are the same. This might be the way to undo isolation, alienation, hatred, fear – and find love.

I am glad this visualization has been provided, as I was curious to know what songs my peers had selected for last week’s assignment, and what implications could be made about our collective mind.

This visualization displays the popularity each song had for the curation. Pieces like the Fifth Symphony, Jaat Kahan Ho, Percussion, Johnny B.Goode, and Melancholy Blues were very popular. I wonder how much this popularity is biased. Could it be that these pieces were selected because of the Fifth Symphony’s high status and recognition in music, the current trends of interest in Indian and Eastern culture (Jaat Kahan Ho), or the familiarity and cultural resonance they evoked in the listeners? (Johnny B.Goode, and Melancholy Blues). I actually selected all of these pieces for my curation (apparently) based on other reasons, but this visualization makes me reflect on how my decisions were most likely biased by my cultural background. Of course, this is only one perspective and way of interpreting the data. We could also argue that these pieces have an essential element that evokes universal acclaim, meaning that they are truly special. I believe it can be seen in both ways and there is truth in both scenarios. This makes me think about ‘best songs of all time’ rankings in which often at the top we’ll find songs like “Imagine” by John Lennon, “Like a Rolling Stone” by Bob Dylan, or “Hey Jude” by The Beatles. It seems like their position is influenced by both biased cultural factors and true specialness.

The visualizations of communities of individuals with similar responses were useful to play with assumptions we could make about how these individuals would agree with one another. While looking at Community 4 (see image below) we can see how these 5 individuals would agree unanimously in several cases. However, it’s interesting to observe how some of them have some choices that don’t meet with any of the rest. Would these be points of disagreement? Could it represent potential conflict in some way? I don’t think this represents an issue in this case, as this curation is a playful exercise. But how about in other contexts such as dating or other kinds of relationship services? Also, It is interesting to think about how these connections are a digital representation of connections that exist in our daily relationships. I can imagine how the individuals who are close to us (particularly friends) represent numerous connections of ideals and preferences. Similarly, the difficulty we encounter in relationships could be seen as isolated (or conflicting) nods.

Of course, the graph only represents one level of agreement, which is the selection of the pieces. However, the rationales for the curation could be quite different, even conflicting. One individual could have selected the pieces based on emotional content, while another could have selected them based on cultural diversity. On the surface, the decision is the same, yet the motive is different. This could create some issues and misinterpretations of data. For example, let’s imagine that two individuals decided to have a celery drink instead of a Coca-Cola. We could make the assumption that both prefer the taste of celery drinks and are in agreement in that area. Yet, on further investigation, one individual might have chosen the celery drink only because of its health benefits and, in fact, doesn’t like the taste of celery at all, while the other individual loves the taste of celery and didn’t consider the health benefits so relevant. I believe this shows how much information can be erroneously interpreted and the potential danger of algorithms, particularly in ethical dimensions.

Image: Community 4
Green: unanimous agreement
Red: representations of possible disagreements

In this linking assignment, I’m going to make connections and analyze differences between the reflections my classmate, James Martin, and I did for Task 8 – Golden Record curation. I was interested in reading James’ reflection and rationale for the curation because he is also a musician. In fact, last semester we were classmates in ETEC 524 and had some interesting discussions about music.

It was very interesting to read James’ rationales as we used different approaches for the selection of the 10 pieces. My selection was guided by the question of how music could provide information about the human experience so that the receptors can understand who we are. For this reason, I focused on selecting tracks that revealed information about our existence as time and space beings, transcendental awareness, the human body, emotions such as joy and fear, our social nature, our intellectual capacity, our creativity, and understanding of the universe. On the other hand, James’ approach was much more focused on the music itself, looking for ways of representing the widest range of musical possibilities. He was very thoughtful in ensuring his curation displayed the “widest possible combinations of diverse musical elements”. For this reason, he included unaccompanied vocal music, instrumental-only music, solo music, accompanied signing and playing, etc.

This contrast between our approaches helps me discover something very interesting. While selecting the pieces, it seems like I didn’t focus too much on the forms of music (I am surprised by this!), but rather on the non-musical content, it is capable of containing, expressing, and transferring. This makes it more obvious to me that music has different dimensions and functions. At one level, it is an art form that creates objects of perception that can be assembled in a variety of ways (hence the richness of music possibility and languages); at another level, it’s a communication tool with immense capacity of revealing implicit content of those who participate in the creation of it.

I believe the approach I followed reveals aspects of my character, as I tend to be very introspective and usually like to explore the deeper and unseen realms of something. However, I think James’ approach was also very skillful and coherent. It would have been very interesting to work as a team for this assignment, as both of our approaches combined could have resulted in a well-founded curation.

As a musician, I found the Voyager Golden Record story to be fascinating. What an incredibly difficult task to choose 27 music pieces out of the millions of songs that have been recorded in history. Even these millions of recordings only represent a small fraction of all the songs created throughout human history.

Carl Sagan states that the objective of the Voyager Golden Record was to choose the songs that best represent humanity. I believe this is an impossible task because the idea of “best” is going to be hugely biased by the cultural background and ideologies of those who select the pieces. Of course, I don’t intend to de-contextualize Sagan’s comment, as he does confess this is a difficult task. However, I think the Voyager Golden Record case is a perfect example of what was stated in this module’s introduction: “what counts as legitimate knowledge is the result of complex power relations and struggles among identifiable class, race, gender, and religious groups” (n.a., n.d.).

With this in mind, I would like to disclose that my curation is highly biased. This is not only due to aesthetic preferences. In some cases, I will use reasoning similar to Sagan’s, who selected some pieces because they represent strong mathematical foundations or philosophical ideas. In my view, these are also biased by ideologies of what is relevant and meaningful content. I actually enjoy how this task exposes the subjectivity of music appreciation and preference, a topic I like to reflect on and always come across while interacting with other musicians.

I think one crucial aspect of doing this task is to think about how the pieces can provide information to the receptors about our human experience. This is the approach I followed when selecting the pieces. Inevitably, I am making some assumptions about the receptors, such as their ability to perceive frequencies through hearing, self-awareness, awareness of a higher universal source, intelligence, and capacity to experience emotions.

Here we go…

(In no particular order of importance)

I’ve selected this piece because rhythm is the most basic form of musical expression. It only deals with localization of sound in time, but it doesn’t have a dimension of tonality (although in this piece some of the drums are tuned) as we would find in melodies in harmonies. The repetitive drum patterns show an understanding of mathematical proportions. This would let the receptors that we are time-beings.
Click here to listen

I’ve selected this piece because of its melodic complexity. This would let the receptors know about our incredible ability to play with organized divisions of space, which in turn also reveals information about the spatial component of our existence and thinking processes.
Click here to listen

I’ve selected this piece because of its vocal component. In my view, the human body is the most complex “gadget” on Earth. With this in mind, the amount of content a human voice (particularly a virtuoso) can transmit is unique and represents the power of our creator.
Click here to listen

I’ve selected this piece because of its spiritual character. In my view, a performance like this is not so focused on the forms of music (and that’s why it is simple) but rather on the silence and the essence from which music (or creations) emerges. This would let the receptors know that we are not just time and space beings, but also spiritual beings. This represents a level of awareness of the formless and timeless being within us – which is probably the same universal being within the receptors
Click here to listen

I’ve selected this piece because of its exuberant and joyful character. This, of course, assumes that the receptors are emotional beings. In my personal view, the purpose of music is to express aliveness, beauty, and joy – to celebrate the fact of existence, which would express an understanding of the value of life.
Click here to listen

I’ve selected this piece because it shows a very important aspect of human life, which is our social nature. The many vocals heard in this piece show our ability to come together as a whole, which reflects an understanding of a level of our existence in which we are not separate and individual.
Click here to listen

I’ve selected this piece because it reflects an outstanding understanding of mathematics. It is very difficult to imagine that the receptors wouldn’t comprehend mathematical foundations – as the universe is built on such principles. I believe this piece would provide a lot of information about our level of intelligence and mental capacity.
Click here to listen

Although all the songs give clues about our technological ability (songs had to be recorded using microphones, for example), this is the only piece that has an electric instrument. This could show the receptors that we also use electrical technology for creative purposes, revealing again our expressive nature.
Click here to listen

I’ve selected this piece because it reveals the dark side of the human mind and soul. This piece evokes fear and I believe it would be important for the receptors to understand our vulnerability and existential struggle. I think it’s probable that the receptors could be beings that have transcended time and space (hence fear) and know what to expect from our behavior under certain circumstances.
Click here to listen

I’ve selected this piece because of its very fractal nature. It consists of a simple phrase that gets modified and elaborated in numerous ways. In my view, this reveals an understanding of how creation unfolds from simple patterns into complex ones.
Click here to listen

In this linking assignment, I’m going to make connections and analyze differences between the reflections my classmate, Christina Hidalgo, and I did for Task 7 – Mode Bending

I think Christina’s mode bend is a good example of what has been mentioned often throughout our course materials about how experiences are always multimodal. The video scribe she produced has visual, textual, auditory, and gestural representations. It was interesting to compare Christina’s approach with mine for many reasons. My mode bend had been mode literal in the sense that I took the objects and did my best to represent them in auditory form (for example the cellphone sounds represent the cellphone image) or evoke them through gestures (for example moving my hands as if I was reading a book represents the Bible). The first couple of seconds I was watching Christina’s video I felt there wasn’t much mode bending going on, as it seemed primarily visual and also included the pictures from the original post. However, I soon begin to realize I was incorrect as I became more attuned to her narration and understood how the representation was largely auditory. Her voice also had a lot of gestures, in the sense that emotions were being transmitted as she expressed ideas and concepts. When this became evident, I closed my eyes and realized that the auditory representation was by itself sufficient as it was very detailed and full of content. In the analysis of my mode bending task, I had observed that my auditory representations were not so clear as the images in the original post because I used sound effects instead of verbal descriptions. Christina’s mode bend supports well this argument, as we can see how using sound to produce words is highly descriptive and precise. This is again pointing to the power of language as one of the most efficient tools for experience description. I could even argue that words through sound (speaking) has a higher degree of content than words through images (writing), as it can transmit emotion with more precision. This is probably why in texting we often use emoticons or other symbols to convey emotion.

I also found it interesting how Christina was able to expand the visual dimension and representation through the use of video. While describing an object many new visual objects would emerge to support the description. This was not necessarily a bend of mode, as it still is a visual representation, however, it seems like the auditory representation gave birth to new auditory representations creating a whole new multi-dimensionality and richness to the descriptions she was making. This also shows interesting dynamics of how information is translated from one form of representation to another (visual to audio) and this creates new possibilities of re-representation of the original source (visual to audio to visual). In effect, it seems in my experience that the different senses are constantly informing one another creating more complex forms of human experience (for example I can hear a song that brings an image to my mind, which then I express in speech, which in turn makes me feel particular emotions that influence my gestures, and so on).

A Multimodal Approach: Gestural + Audio

For this task, I decided to do a mode-bending that included two new modes not used originally for task 1: gestural representation and audio representation. Originally I was going to do only a gestural representation but then I thought that including an audio representation would allow me to give more clues to the viewer, as I am presenting the objects without giving away their identity through words (although there is a link to reveal that information).

Using gestures, I am expressing how each object is used. Using sound, I am expressing what sounds can be produced with the objects or activities I do with the objects. It is interesting to contemplate how, compared to the original task, there is no visual nor textual representation of the objects, and how the immediacy of identifying them is reduced. This made me think about how visual and textual representations are usually the most precise modes to describe objects. Seeing an object and its name makes possible instant recognition. I believe it mirrors how the mind works, as it seems in my experience that, in thinking, a name can’t be separated from form (try to imagine a nameless form!). However, what I also learned from this mode-bending task is theat audio (not as speaking, but sound effects) and gestures are very efficient to illustrate actions or dynamic content. This made me think about how when we think about verbs, the images produced in the mind are dynamic and, in that sense, more like a gesture or a sound effect.

Hence, this redesign process was one of going from static content to dynamic content; from objects to activities; from nouns to verbs. However, if I look at it more critically, it seems like the act of representation is in itself a dynamic process of transformation. Even in the case of the images and the words, there is a initial process in which the objects (which is a perceptual experience) are converted into a static form (pictures and words). This help me to understand Kalantzis and Cope’s (2009) observation that “all forms of representation, including language, should be regarded as dynamic processes of transformation rather than processes of reproduction” (p.175). Another important idea expressed by Kalantzis and Cope’s (2009) and in ETEC 540 is that experiences are multimodal. For example, the gestural representation was clearly visual as well, although gestures could also be expressed only through sound or words. But could we have gesture separated from a visual, auditory, or linguistic experience?

Another interesting observation that came from this task is that my gestural and visual representations are biased. To read these visual and gestural texts the observer must have a cultural background sufficiently close to mine. An individual from 2000 years ago wouldn’t be able to understand most (if not all) of the representations. The gestures and sounds are informed by cultural objects that belong to a particular culture and time. However, it is interesting to think about how some of the emotions I display in the videos (for example enjoyment or peace) have a transcendental quality that could be understood by any human. This made me think about Dobson & Wilinsky’s (2009) assertion that “literacy should be viewed, rather, as a set of complex characteristics and processes that influence and are influenced by social context and personal circumstance” (p.15). From my analysis of how the objects biased the representations, I can see the truth in this statement. However, from the analysis of the transcendence of some texts (for example emotions), I believe it could be argued that maybe there is some literacy that is inherent in human beings, regardless of context or culture.

Object 1

Reveal Object 1

__________________________________________

Object 2

Reveal Object 2

__________________________________________

Object 3

Reveal Object 3

__________________________________________

Object 4

Reveal Object 4

__________________________________________

Object 5

Reveal Object 5

__________________________________________

Object 6

Reveal Object 6

__________________________________________

Object 7

Reveal Object 7

References

Dobson, T. M., & Willinsky, J. (2009). Digital literacy. (pp. 286-312). Cambridge University Press.

Cope, B. & Kalantzis, M. (2009). Multiliteracies: New literacies, new learning. Pedagogies: An International Journal, 4:3, 164-195, DOI:Links to an external site.
10.1080/15544800903076044Links to an external site.