Visual Transformation Through Creative Semantics
Created by Marc Aubanel, Mike Singh & Kimberly Wagner
The art and technology studioSoso Limited, complete innovative projects that “incorporate elements of dynamic typography, video manipulation, computer vision, sensor technologies and sound design” (Soso Limited, n.d.a). Their dynamic text presentation, Reconstitution 2008 (Mead, n.d.;Soso Limited, 2008), was a live audiovisual remix of the Presidential debate. They designed the software to analyse the audio and video and represent it through dynamic text. Their project Semantic Sabotage is an open source platform “for creating live typographic YouTube transformations” (Soso Limited, Semantic Sabotage, n.d.b). It is possible to experiment with a variety of transformations in the gallery, and to get the source code to make your own tool. We had the idea of creating our own text transformation tool, but due to logistical barriers decided to create a fixed dynamic text presentation to present our findings.
There is little research specifically on dynamic text presentations. There were three studies that were more directly relatable: one, a study on designing efficient text presentations (with multimedia) for improved memory (Huang & Lin, 2001); two, a graduate thesis study of one prosody-enriched dynamic text presentation technique to determine if it enhanced the reading of electronic text (Marks, 2009); and, three, an eye movement study to determine if reading on a mobile device using the rapid serial visual presentation (RSVP) format instead of the page format (Oquist, Hein, Ygge, & Goldstein, 2004). There is also relevant relatable research in the area of representing emotion with animated text (Rashid, 2005), emotive captioning (Lee, Fels, & Udo, 2007), and improving closed captioning for the hearing impaired through dynamic captioning (Hong, Wang, Xu, Yan, & Chua, 2010). We discovered that dynamic text presentations can increase reading speed and comprehension if the method is focused and not distracting. The use of semantics can improve understanding and add meaning visually.
Walter Ong (1982), outlines the history of orality and the development of print literacy which provides the background and foundation for paper text forms. He also develops the concept of “secondary orality” which refers to new oral mediums that reach large audiences, but are still carefully controlled like print. New digital mediums are less controlled and in the hands of individual users. In Bolter’s (2001, Ch. 4) Writing Space, he does not refer directly to the term dynamic text but the umbrella term “hypermedia” which integrates both image, sound and text. He states, “In presenting animation and digitized video, a webpage can supplement or bypass prose altogether” (Bolter, p.47). Our video strives to show the effectiveness of dynamic text. Our topic dynamic text represents writing breaking free from its static, fixed position on a printed page as it becomes capable of movement, flexible-positioning, shape, image, and hyperlinking. It is a part of the new digital medium for reading and writing and the building on past developments in writing (Dobson & Willinsky, 2009).
Bolter (2001, Ch. 4) also outlines the concept of “remediation” when discussing the ever changing concept of print and image. Initially, print always took precedence and control over the image. Traditionally, print was produced by one means and pictures were done by another. Through the use of the computer, this is no longer the case. Authors now can easily incorporate both text and image which gives them a new found freedom of expression. He advocates that we now live in a visually dominant culture, and that even in printed media, images are changing the way prose is presented. Ekphrasis and reverse ekphrasis in terms of new media designers’ urge to “redefine the balance between word and image” (Ch. 4) is described by Bolter; we extend this idea to a ‘semantic ekphrasis’ meaning to make the words themselves visually meaningful in shape, colour and style.
Landow (2006) in Hypertext 3.0 states, “The code-based existence of electronic text that makes it virtual also makes it infinitely variable. If one changes the code, one changes the text (p. 36). There is code sitting invisible behind the dynamic text we see on the screen which makes this visual text possible. Design elements are no longer in the domain of the typesetter but in the hands of the author. There is a long history of visual embellishment on paper, but now anyone has the capability of “manipulating visual effects” (p. 88). Animated text “entirely controls the reader’s access to information at the speed and at the time the author wishes” (p. 93). Dynamic text is a combination of modes: written language, visual representation, and spatial representation. The multiliteracies (Cope & Kalantzis, 2009; New London Group, 1996) concept calls for an expanded understanding of what it means to be literate. Dynamic text requires active creating and viewing as a person makes sense of its semantics.
This centrality and importance of the image is also a recurring theme in many of the web 2.0 platforms showcased in our video. Alexander (2013) states that “the open nature of Web 2.0 platforms, connected by hyperlinking, lets learners pursue connections across multiple lines of thought … multiple browsers or tabs within a browser lets learners pursue multiple inquiries in rapid and almost sequential sequence” (p.156). This idea of pursuing multiple connections at a rapid pace is exemplified in the visual representations in both Gap Minder and Visible Tweets. With a few keystrokes, one can see a visual representation of data which makes it much more effective that the written word itself. These web applications also expound the concept of transforming data into visualizations extends the concept of word visualization being examples of searching a large data set and then representing that text and information dynamically. Data visualizations, such as the MIT Health Infoscape, represent a large set of data presented in a way that people can understand (PBS, 2013). David Talbot (2013a; 2013b) of MIT Technology Review describes and demonstrates the frequency of a tweeted word in sped up video to show how a new software chip has been developed to allow real time display of data from a large data set. Memes, which are the ultimate combination of image and text, use multi-media to look at things from a humorous standpoint. Although they may seem quite simple, their viral nature demonstrates their lasting impact.
We devised the following thesis for our multi-media compilation:
Text is no longer constrained by its physical tools and mediums. Hypertext allows for new methods of representation as we build on traditional text forms within digital environments.
Sub-Topics:
-
Changing relationship between word and text
-
Visualization online tools
-
Kinetic Typography
-
Hypertext and linking
-
Memes
-
RSVP
We planned our dynamic text presentation video in Google Spreadsheet indicating the segment, text, images, animations, and sounds. We project planned in a Google Document, meeting there and in Skype once a week, and communicating in the document as needed. Marc decided that due to limited time, that we would create a linear, non-interactive piece. Using a combination of Adobe After Effects and Premiere, he challenge himself to create animated text and visuals. Kim learned how to use Powerpoint to create a dynamic text presentation using PowerPoint Spice (2012; 2013) tutorials as the files could be finalized as video. Kim and Marc completed video screen captures to demonstrate several online visual text tools. We brainstormed ideas for how we could give words a visual aspect, like the brain in ‘cognitive’ and the conversion of the words ‘image,’ “emotion,’ ‘transformed,’ and ‘visually’ being inspired by the Visual Text Project (2012) and Ji Lee’s dynamic word e-book (2011; n.d.). Mike completed a storyboard for his segments and Marc created them for him. We reviewed 11 video drafts during the process. An open source, free to use, background music was chosen to compliment the content of the presentation. We made a decision to not include direct references to the reading in our artistic product because we wanted the dynamic text techniques to creatively inform and enlighten the viewer.
Bibliography
Alexander, Bryan. (2013). Web 2.0 and emergent multiliteracies: Theory into Practice. 47(2), pp. 150-160. Retrieved November 23, 2013, from http://www.tandfonline.com.ezproxy.library.ubc.ca/doi/pdf/10.1080/00405840801992371
Bolter, Jay David. (2001). Writing space: Computers, hypertext, and the remediation of print [2nd edition]. Mahwah, NJ: Lawrence Erlbaum. Retrieved November 1, 2013, from http://www.adamjrichards.com/the_death_of_rock_n_roll/resources/html_format/Bolter_-_Writing_Space.html
Cope, B., & Kalantzis, M. (2009). “Multiliteracies”: New Literacies, New Learning. Pedagogies: An International Journal, 4(3), 164-195. Retrieved November 1, 2013, from http://search.ebscohost.com.ezproxy.library.ubc.ca/login.aspx?direct=true&db=ehh&AN=43440672&site=ehost-live&scope=site.
Dobson,T. & Willinsky, J. (2009). Digital Literacy [chapter, preprint version]. The Cambridge Handbook of Literacy. Cambridge: Cambridge University Press. Retrieved November 5, 2013, from ETEC 540 Course Content.
Hong, R., Wang, M., Xu, M., Yan, S., & Chua, T. (2010). Dynamic captioning: video accessibility enhancement for hearing impaired. MM ‘10 [Proceedings of the international conference on Multimedia], pp. 421-430. Retrieved November 1, 2013, from http://dl.acm.org.ezproxy.library.ubc.ca/citation.cfm?id=1874013.
Huang, S. & Lin, M. (2001). Designing efficient text presentation of multimedia CAI – the evaluation of dynamic text patterns and the negative repetition effect on memory. Computers & Education, 37(2), pp. 127-140. Retrieved November 1, 2013, from http://www.sciencedirect.com.ezproxy.library.ubc.ca/science/article/pii/S0360131501000434.
Landow, G. (2006). Hypertext 3.0: Critical Theory and New Media in an Era of Globalization. Baltimore: The John Hopkins University Press.
Lee, D., Fels, D., & Udo, J. (2007). Emotive captioning. Computers in Entertainment (CIE), 5(2), article 11, pp. 1-15. doi: 10.1145/1279540.1279551.
Lee, J. (2011, Oct 6). Word as Image [visually animated words; video]. YouTube. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=J59n8FsoRLE.
Lee, J. (n.d.). Word as Image [website]. URL redirects to Facebook page. Retrieved November 5, 2013, from http://wordasimage.com.
Marks, E. (2009). A Prosody-Enriched Dynamic Text Presentation Technique for Enhanced Reading of Electronic Text. A thesis submitted to the Faculty of the Graduate School at Howard University. Retrieved November 1, 2013, from http://www.academia.edu/435771/A_Prosody-Enriched_Dynamic_Text_Presentation_Technique_for_Enhanced_Reading_of_Electronic_Text.
Mayer, R. (2009). Multimedia Learning. 2nd Edition. Santa Barbara, California: Cambridge University Press.
Mead, D. (n.d.). Motherboard TV: Sosolimited Video-Hacks the Presidential Debate [article and video]. Motherboard. Retrieved November 1, 2013, from http://motherboard.vice.com/blog/motherboard-tv-sosolimited-video-hacks-the-presidential-debate.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Educational Review, 66(1), 60-92. Retrieved November 5, 2013, from http://wwwstatic.kern.org/filer/blogWrite44ManilaWebsite/paul/articles/A_Pedagogy_of_Multiliteracies_Designing_Social_Futures.htm.
Ong, W.J. (1982). Orality and Literacy: 30th Anniversary Edition (2012). Forwarded by J. Hartley. London & New York: Routledge.
Oquist, G., Hein, A., Ygge, J. and Goldstein, M. (2004). Eye Movement Study of Reading on a Mobile Device Using the Page and RSVP Text Presentation Formats. Mobile human-computer interaction–MobileHCI 2004: 6th international symposium, Glasgow, UK, pp. 108-119. Retrieved November 1, 2013, from http://download.springer.com.ezproxy.library.ubc.ca/static/pdf/640/bok%253A978-3-540-28637-0.pdf?auth66=1384055128_3739773034e7c22a52530be4ab4051df&ext=.pdf.
PBS. (2013, May 9). The Art of Data Visualization . Offbook series production. Retrieved November 20, 2013, from http://video.pbs.org/video/2365039781/.
PowerPoint Spice. (2012, Dec 13). How to Make Kinetic Typography Effects – Advanced Powerpoint Animation Tutorial. YouTube. Retrieved November 5, 2013, from https://www.youtube.com/watch?v=kI39o2K3KD4.
PowerPoint Spice. (2013, Jul 26). Amazing Motion, Fire, and Shiny Text Effects in PowerPoint – Advanced Animation Tutoriall. YouTube. Retrieved November 5, 2013, from https://www.youtube.com/watch?v=ITXGGEMdpE0&feature=c4-overview&list=UUIWglTtslnzN3rSEUbldzag.
Rashid, R. (2005). Representing Emotions with Animated Text. A thesis submitted to the Faculty of Information Studies at University of Toronto. Retrieved November 1, 2013, from http://search.proquest.com.ezproxy.library.ubc.ca/docview/250989857.
Soso Limited. (2008). Reconstitution 2008 . Retrieved November 1, 2013, from http://sosolimited.com/reconstitution/.
Soso Limited. (n.d.a). Info. Retrieved November 20, 2013, from http://sosolimited.com/info/.
Soso Limited. (n.d.b). Semantic Sabotage [web tool]. Retrieved November 2, 2013, from http://sabotage.sosolimited.com/.
Talbot, D. (2013a, Oct 18). Graphics Chips Help Process Big Data Sets in Milliseconds. MIT Technology Review. Retrieved November 1, 2013 from http://www.technologyreview.com/news/520021/graphics-chips-help-process-big-data-sets-in-milliseconds/.
Talbot, D. (2013b, Oct 18). Visualizing Big Data in Milliseconds on Cheap Computers . MIT Technology Review. Retrieved November 10, 2013, from http://www.technologyreview.com/video/520481/visualizing-big-data-in-milliseconds-on-cheap-computers/.
Visual Text Project. (2012). Visual Text Project [website]. Retrieved November 12, 2013, from http://www.visualtextproject.org/.
Multimedia Credits
Aubanel, M. (2013). Facebook page. Retrieved November 10, from https://www.facebook.com/marc.aubanel.75.
Apple Calendar App. (2013). Marc Aubanel’s Apple Calendar [video_capture].
Cheezburger. (n.d.). Cheezeburger Cat [screen capture]. Retrieved November 9, 2013 from https://i.chzbgr.com/maxW500/1658715904/h7CB1AF70/.
Cheezburger. (n.d.). Samuel L. Jackson [screen capture]. Retrieved November 18, 2013 from https://i.chzbgr.com/maxW500/7893491200/h17B2C038/.
Cheezeburger Fail Blog. (n.d.). Sail Fail [screen capture]. Retireved November 17, 2013, from http://failblog.cheezburger.com/.
Free Great Picture. (n.d.). HD colour background wallpaper 18429 [image]. Retrieved November 10, 2013, from http://www.freegreatpicture.com/background-color-theme/hd-color-background-wallpaper-18429.
Free Great Picture. (n.d.). High-resolution color background 14928 [image]. Retrieved November 21, 2013, from http://www.freegreatpicture.com/background-color-theme/high-resolution-color-background-14928
funny junk. (n.d.). Ermahgerd Kerrets [screen capture]. Retireved November 15, 2013 from http://www.funnyjunk.com/funny_pictures/4125604/Ermahgerd.
Gapminder. (n.d.). Gapminder World: Health and Wealth of Nations [video_capture]. Retrieved November 16, 2013, from http://www.gapminder.org/.
MIT Senseable City Lab. (2011). Health InfoScape web page [video_capture]. Directed by Carlo Ratti. Collaboration with General Electric. Retrieved November 20, 2013, from http://senseable.mit.edu/healthinfoscape/.
Ong, W. (1982). Chapter 1. Ong’s Orality and Literacy: 30th Anniversary Edition (2012) [screen capture]. Retrieved November 2, 2013, from http://lib.myilibrary.com.ezproxy.library.ubc.ca/Open.aspx?id=1960.
Open Clip Art Library. (2011, May 7). Fish forming letter m [image]. Retrieved November 10, 2013, from http://openclipart.org/detail/137053/fish-forming-letter-m-by-moini.
Open Clip Art Library. (2010, Apr 16). Light Bulb On [image]. Retrieved November 10, 2013. http://openclipart.org/detail/48397/light-bulb-on-by-palomaironique.
Spell with flickr. (n.d.). Transformed [image]. Created and retrieved November 15, 2013, from http://metaatem.net/words/transformed.
Spell with flickr. (n.d.). Visually [image]. Created and retrieved November 15, 2013, from http://metaatem.net/words/visually.
Spreeder. (2013). Web app demonstration [video_capture]. Retrieved on November 19, 2013, from http://spreeder.com.
Team Little Guy. (2008). I Gotta Have More Cowbell!! [screen capture]. Retrieved November 12, 2013, from http://teamlittleguy.files.wordpress.com/2008/08/more-cowbell-tshirt.jpg.
The Atlantic. (1945). ‘As We May Think’ article [video_capture]. Retrieved November 15, from http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/.
Thinkmap Visual Thesaurus. (2013). VocabGrabber web page [video_capture]. Retrieved November 16, 2013, from http://www.visualthesaurus.com/vocabgrabber/#.
Tweet Beam. (n.d.). Tweet Beam web page [video_capture]. Retrieved November 16, 2013, from http://www.tweetbeam.com/.
Twistori. (n.d.). Twistori web page [video_capture]. Retrieved November 16, 2013, from http://www.twistori.com/.
Tsyolin. (2012, Apr. 7). Tsyolin Befriends Wooden Toaster Overture [instrumental music]. YouTube. Retrieved November 21, 2013, from https://www.youtube.com/watch?v=D6Jfctt1b8o.
Video Copilot. (2013). Vintage paper textures [image]. Retrieved November 9, 2013, from http://www.videocopilot.net/blog/2013/08/vintage-paper-textures/.
Visible Tweets. (n.d.). Visible Tweets web page [video_capture]. Retrieved November 16, 2013, from http://visibletweets.com/.
Wolfram Alpha. (n.d.). Wolfram Alpha computational engine [video_capture]. Retrieved November 16, 2013, from http://www.wolframalpha.com/.
Appendix A: Dynamic Text Presentation Examples
Beaudoin, O. (2007, Apr 19). Typolution [pollution text animation; video]. YouTube. Posted by graphkzr1234. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=zVPfTlpCKaw&noredirect=1.
Canal, C. & Baptista, S. (2011, Jun 14). Im Gonna Make it Better (After Effects Animation) [text animation; video]. YouTube. Posted by Manuel Garcia. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=p5_WkM7S5U0.
Dover, B. (2010, Jan 8). Emotional Lines Kinetic Typography [kinetic typography; video]. YouTube. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=ABsNUoA3g-0.
Heaton, J. (2008, Oct 15). Kinetic Typography – Everywhere You Look (Full House) [kinetic topography; video]. YouTube. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=7onvQf9IC1M.
jessportfolio. (2009, May 5). hypertext [concert text animation; video]. YouTube. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=GCfs9nOMXIE.
mgicboy18. (2013, Jan 6). What teachers make (Kinetic Typography) [poetry kinetic typography; video]. YouTube. Poem and performance by Taylor Mali. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=t-FUTp_oO5s.
Shuster, E. (2009, Oct 14). Helvetical in Motion – Kinetic Typography [kinetic text presentation; video]. YouTube. Posted by solidus1985. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=R1ZBknDPlu4.
Wesch, M. (2007, Mar 8). The Machine is Us/ing Us (Final Version) . YouTube. Retrieved November 1, 2013, from https://www.youtube.com/watch?v=NLlGopyXT_g.
