Text spaces in Augmented Reality (AR)

Oren Lupo
ETEC 540b
Dec 1, 2010

Augmented Reality (AR) is a field of computing and visual simulation that blends real-world environments with virtual, computer-generated elements such as text data, visual reference points, animations and 3D graphics.  Whereas virtual reality replaces the real world with a purely virtual one (e.g. Second Life), AR brings the two together, creating an immersive experience that is perceived by our brains as an enhanced or supplemented view of the world. (O’Shea, 2009)

The virtual elements in AR technologies are connected or mapped onto real locations in three-dimensional space. Hence a key metaphor in the discourse of AR is the idea of using computer technology as a medium to “overlay” information on top of or within a display of the physical world. (Manovich, 2002) Another aspect of AR technologies is the interaction between virtual layers and real-world environments in real time (Azuma et al, 2001). Since the AR experience is interactive and dynamic—that is, things that happen in the real space seem to affect the virtual layer—the effect on the display screen is one of a context sensitive user interface that combines the virtual with the actual.

A typical AR system set up would include a webcam, computer or mobile device such as a smartphone, and software that detects the position of one or more visual markers in the webcam’s view of the environment. The system orients the virtual elements relative to the position, shape, design or other physical features of the identified surface marker. This makes it possible for virtual elements to appear to respond and interact with all kinds of real-world surfaces, signs, shapes, or objects by tracking their marker’s position. A 3D virtual model superimposed in this way may tilt, rotate, or change its perspective and lighting as its corresponding real-world surfaces change their positions, textures or colours. (Zhou et al, 2008) (see Fig. 1)

Fig. 1 Demonstration of GE Electric Smart Grid Wind Turbines in AR [visit website]

The virtual elements that can be presented by AR technologies are not limited to visual ones; they may be sounds, physical sensations like vibrations, or even smells. Users can interact with the virtual layer by using a keyboard, mouse or voice commands; by changing the markers that are in the camera’s view; or in other ways, such as with computerized face detection.

A second type of input currently in use with AR systems is spatial rather than visual. It works with GPS (global positioning system) navigation that tracks the three-dimensional locations (latitude, longitude, and altitude) of objects and landmarks relative to the positions of space satellites orbiting the Earth. The viewing platform for this is typically a mobile smartphone. The phone’s mobile computer represents GPS data with text and 3D virtual objects that are overlaid onto a handheld display. This creates a dynamic mixed-reality experience on a larger scale outdoors, where basic directional data from a compass and distance markers can also be added.

This type of AR does not yet recognize natural features from the phone’s video camera, but this is a technological hurdle that is likely to be overcome soon. (Zhou et al, 2008) The advantage of this second type of mobile AR technology is that it can superimpose information in the smartphone display on live locations—for example, a landmark building on a city tour—by using geotags that are referenced to one or more active online databases. (Colwell, 2009). Such AR browser applications are already on the market. For example, Google Maps points-of-interest, Wikipedia entries, photo-,sharing sites like Flickr, or social-network review websites, all can be shown with AR browsers for smartphones, such as junaio and layar. (see Fig. 2)

Fig 2. Screen capture from Yelp Monocle mobile app for finding restaurants

AR systems also have many applications in the medical, industrial and scientific fields, where augmentation can be used to visualize, model and explain processes and structures without actually changing anything about what is being observed. For example, medical imaging can be used in training and diagnostics to visualize physical systems and organs in a patient’s body, and to train students in the use of tools and techniques. (See Fig. 3)

Fig. 3 From Computer Assisted Medical Procedures Institute at the Technische Universitat Munchen

Industrial assembly, maintenance, and repair of complex machinery likewise can be done with a virtual overlay that moves hard-to-see parts and systems in and out of position with a webcam or heads-up display over an actual device. AR provides the option of supplementing each step with many levels of instruction, different presentation styles, multilingual speech and texts, various analytic activities and real-time assessments for trainees. Other areas where AR technologies have been adopted are in architecture and urban design, advertising, magazines, travel guides, product displays, real estate and video games. (Innovation Forum, 2010)

AR in education:  the augmented book

The two types of AR technology outlined above—one developed for geographical spaces and large-scale locations using mobile devices, and the other for table-top and enclosed contexts using webcams and computers—are currently two different but related areas of research and practical technological development. (Zhou et al, 2008) This division will become less important in a few years time. However, in terms of the potential for discovering and realizing educational uses for AR, these two current technological paths are bound to affect how AR is actually used in the classroom.

Functional and technological accessibility is not usually the main factor in deciding whether to adopt a certain educational technology into practice. Taking a deep approach to learning and student-centred teaching, along with outcomes-centred curriculum design, and all kinds of other institutional considerations, are sure to be more relevant than technological capability in deciding how AR might make its way into educational settings. (Phillips, 2009) On the other hand, AR does hold out the prospect of providing rich contextual, in situ learning experiences with present-day technologies. For example, augmented learning-exploration and discovery games may be devised to enhance student field trips to museums or historical sites, assuming that they are outfitted with mobile devices. Simplified versions of AR training models also could serve in classroom and individual activity-based explorations of the connected nature of information in the real world. (Johnson et al, 2010)

An important feature of the type of AR highlighted in this Course Assigment, which is also the third main area of educational AR research, is the use of the traditional paper book as an interface for augmentation.  In terms of the display and manipulation of text on the flat page, the possibilities of the augmented book are equally open to innovation in typography, graphic design and 2D animation as they are to eye-catching design in the 3D domain.

One outstanding example of this may be seen in the thesis project “Jekyll and Hyde Augmented Reality Book” created by students Marius Hügli and Martin Kovacovsky at HGK Basel. (see Fig 4.) Another example in the same vein is the Magic Book installation created by designer Camille Scherrer.

Fig. 4 This is a live video recording of the AR book with the webcam mounted into the desk lamp, displayed on the desktop iMac computer

As visually fascinating a such demonstrations of book-based AR are to watch, the main advantage of taking this path toward putting AR  technologies into service in education is not primarily technological. While the technical possibilities for superimposing computer-generated information and visual elements on nearly any surface or within any space are growing all the time, the central place of paper books in educational culture makes them the best choice for introducing AR as a practical learning technology (Yamada 2009). This has much to do with the affordances of print books as technologies for organizing and displaying information.

As Walter Ong (1982) has pointed out, printed books engender a “meaningful space” for words and texts that is both highly ordered and endlessly flexible within the limits and conventions of typography, printing, and illustration (Ong, p. 129). Ong adds that modern printed books are “closed” (pg. 131) technologies compared to the manuscript codex, in that printed books are formally complete, self-contained, and reproducible as identical copies. It is interesting to consider where future AR books might fall within this dichotomy of open/closed technologies for organizing information spatially.

Bolter (2001) compares the closed quality of printed books with e-books, which are delivered with purpose-built, networked devices called e-book readers (e.g., the Amazon Kindle). Bolter observes that e-books represent a heterogeneous and “hybrid” form of paper books (p. 79-81). E-book readers convert printed text into hypertext, but they intentionally retain many features of traditional printed books in their design as well. They also add further capabilities that printed books don’t have, such as a search function, user-selected layout and typography, and access to vast online portals for e-bookselling, reference works and libraries. These features, according to Bolter, are part of a remediation of the printed book away from its fixed physical structures (e.g. the index) and its notional quality of closure. However, the guiding principle for e-book readers is still to re-make the experience of reading paper books, rather than simply serving a secondary function as tablet computers. AR books make possible another type of “hybridized” enhancement to print, though it is one that may be closer to hypertext than to the reconfiguration of the traditional printed book that we see with e-books.

AR books, or at least the few that have been published recently, have a taken a different approach than e-books to that which Ong calls the “spatializing of the word” (p. 133). We see today AR books that feature animated, multimedia 3D scenes that are integrated into their storylines using some variation of the previously described marker technology (for a detailed overview of this type of book, see the companion section to this assignment, Rip.Mix. Feed). The design of AR books seen so far doesn’t weave 3D virtual enhancements into the narrative content of the books, other than simply adding visual appeal. The current state of the art in AR software development is “marker-less tracking”, which will make it possible for almost any surface image or object on an AR book page to be recognized as a launch point for virtual elements and embedded with digital information (Kim et al, 2009; Dias, 2009). Once this happens, it will be much easier to work effects such as those we see in the “Jekyll and Hyde Augmented Reality Book” into printed texts, and achieve a broader range of subtle and startling effects. But this is still a prospect for the future in terms of commercial publication.

One of the reasons that AR has made its first inroads in the children’s book market, with titles such as author Tony DiTerlizzi’s The Search for WondLa, is the marvellous novelty of seeing the 3D graphics embedded in a fanciful story along with 2D illustrations.  Books like this one sometimes also have limited keyboard or mouse interactivity. This allows the 3D graphics and animations to appear as if the page is a miniature stage where the action is taking place, and to respond to cues from the viewer. This attribute could make AR books more like enhanced board games once the interactive capability is perfected and includes voice commands or gestures. However, the narrative and informational construction of books lends itself to the physical action of page turning—which is also something we typically see as a feature of e-book readers—and the portability and compactness of books cannot be duplicated with flat board games. For these reasons, it seems to me that AR books will continue to retain a fairly conventional bound paper-quire design.

If AR books do catch on more widely, especially in the domain of classroom textbooks, there will likely be a change in the way that tangible books are regarded in terms of their imposition of closure, both formal and informational.  This would correspond to a shift in the way we view the printed page as type of display space. AR books offer the possibility of bringing virtual elements—textual or visual—into the visual scene of the page, and into the narrative construction formed by the text itself. Whether appearing in print as marks made with ink, or as superimposed digital characters, the content of the printed book is shifted into the electronic domain of hypertext when AR is used for narrative purposes. All electronic texts are said to have a dimension of “hypermediacy” (Bolter, p. 185) because they can go further than what is on the page in enabling their references. They can activate elements that are outside the formal structure of the page by linking pathways to different media, thus changing the pattern, experience and order of reading. AR enhanced books likewise enable the reader to activate links within the page—again, either as part of the printed text or the visuals—and then import data, images, sounds, or animations, etc., from online sources and superimpose them onto the page. This could greatly change the experience of reading texts into a mediated one without materially changing the interface of the printed book.

For example, a single children’s AR book might not feature only one set of 3D illustrations; it could have hundreds of variations depending on the age of the child reading the book. The same might go for a multilingual edition of a book that teaches languages, using a hybrid layout composed of print and virtual spaces in different languages that are practically interchangeable with this sort of technology. The printed page would then become a new kind of “hybrid” display interface, while at the same time retaining the structure and affordances of a traditional printed book.

This brings to mind some reflections put forward by N. Katherine Hayles (2003) on natural languages and the capabilities of computer technologies in the domain of “Electronic Literature”. Hayles observes that writing for computerized contexts has often responded to the mediated situation of its production and display—that is, authors using the tools of language and of electronic media design together to create their works. In the case of electronic literature we have seen so far, this has mostly been on video displays or projection screens. Hayles’s argument is that the effect on writing and reading created through electronic media is not simply a creative strategy or change in discourse on the part of the author. It is an effect that imposes new “rhetorics, grammars, and syntaxes unique to digital environments” onto the practices of writing and reading.

But in the case of AR books, this effect is complicated by the fact that we are not confronted with a video display screen as the primary surface on which writing takes place. The immersive effect of the AR blended-environment is not “on” the page, nor is it “on” the display screen of our computers or smartphone. It is, to use the terminology of AR, an “overlay”, an imposed layer, which is distinct from the tangible surface of the printed page, but which is not computer-generated in a way that we’ve come to expect from using display monitors and projectors. To my mind, there is little doubt that trying to work out, as Hayles puts it, the “new modes of attending” demanded by AR books will be a challenge for media theorists. It may be that for the use of AR books in educational contexts to really succeed, it will have to rely some of the insights to come in media theory in order to get beyond the conventional formats of 3D-enhanced pop-up books and augmented encyclopedias.

.

References

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent Advances in Augmented Reality. IEEE Computer Graphics and Applications, 21(6) Retrieved from http://www.cs.unc.edu/~azuma/cga2001.pdf

Bolter, J.D., (2001). Writing Space: Computers, hypertext and remediation of print. New York: Routledge.

Colwell, S. (October 8, 2009) GPS-Based Augmented Reality. GPS World. Retrieved from http://www.gpsworld.com/consumer-oem/handheld/gps-based-augmented-reality-9022

Dias, A., (2009). Technology Enhanced Learning and Augmented Reality: An Application on Multimedia Interactive Books. International Business & Economics Review. 1(1). Retrieved from http://revistas.ulusofona.pt/index.php/iber/article/view/862/699

Hayles, N. K., (2003). Deeper into the Machine: The Future of Electronic Literature. Culture Machine, 5. Retrieved from http://www.culturemachine.net/index.php/cm/article/viewArticle/245/241

Innovation Forum (January 15th, 2010) Augmented Reality, immersive cutting edge Technologies. Retrieved from http://www.innovauoc.org/foruminnovacio/en/2010/01/10-forum-innovacio-realitat-augmentada-tecnologia-immersiva-avantguarda

Johnson, L., Levine, A., Smith, R., & Stone, S. (2010). Relevance for Teaching, Learning, or Creative Inquiry. In The 2010 Horizon Report. Retrieved from http://wp.nmc.org/horizon2010/chapters/simple-augmented-reality/

Kim, K., Park, J., and Woo, W. (2009) Marker-Less Tracking for Multi-layer Authoring in AR Books. Lecture Notes in Computer Science. 57. Retrieved from http://icserv.gist.ac.kr/mis/publications/data/2009/fulltext_2.pdf

Manovich, L. (2005). The Poetics of Augmented Space: Learning from Prada. Noemalab. Retrieved from http://www.noemalab.org/sections/ideas/ideas_articles/pdf/manovich_augmented_space.pdf

Ong, Walter. (1982). Orality and literacy: The technologizing of the word. London: Methuen.

O’Shea, P. (Apr 9th, 2010). What Does Educational Augmented Reality Mean? Educators’ Royal Treatment. Retrieved from http://www.educatorsroyaltreatment.com/2010/04/09/what-does-educational-augmetned-reality-mean/

Phillips, R. (2005). Pedagogical, Institutional and Human Factors Influencing the Widespread Adoption of Educational Technology in Higher Education. In H. Goss (Ed.), Balance, Fidelity, Mobility: Maintaining the Momentum, Proceedings of the 22nd ASCILITE Conference. Retrieved November 20, 2010, from http://www.ascilite.org.au/conferences/brisbane05/blogs/proceedings/62_Phillips.pdf

Zhou, F., Duh, H.B.L., Billinghurst, M. (2008). Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR. International Symposium on Mixed and Augmented Reality, 2008. ISMAR 2008. 7th IEEE/ACM. Retrieved from http://ir.canterbury.ac.nz/bitstream/10092/2345/1/12613246_2008-Trend-inAugmentedRealityTrackingInteractionandDisplayAReviewofTenYearsofISMAR.pdf