THE AUGMENTED REALITY


ugmented reality is a term for a live direct or indirect view of a physical real-world environment whose elements are augmented by virtual computer-generated imagery. It is related to a more general concept called mediated reality in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality. In the case of Augmented Reality, the augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally usable. Artificial information about the environment and the objects in it can be stored and retrieved as an information layer on top of the real world view. The term augmented reality is believed to have been coined in 1990 by Thomas Caudell, an employee of Boeing at the time. Augmented reality research explores the application of computer-generated imagery in live-video streams as a way to expand the real-world. Advanced research includes use of head-mounted displays and virtual retinal displays for visualization purposes, and construction of controlled environments containing any number of sensors and actuators. There are two commonly accepted definitions of Augmented Reality today. One was given by Ronald Azuma in 1997. Azuma’s definition says that Augmented Reality combines real and virtual is interactive in real time is registered in 3D Additionally Paul Milgram and Fumio Kishino defined Milgram’s Reality-Virtuality Continuum in 1994 . They describe a continuum that spans from the real environment to a pure virtual environment. In between there are Augmented Reality (closer to the real environment) and Augmented Virtuality (is closer to the virtual environment).
Mediated Reality continuum showing four points: Augmented Reality, Augmented Virtuality, Mediated Reality, and Mediated Virtuality on the Virtuality and Mediality axesThis continuum has been extended into a two-dimensional plane of “Virtuality” and “Mediality”. Taxonomy of Reality, Virtuality, Mediality. The origin R denotes unmodified reality. A continuum across the Virtuality axis V includes reality augmented with graphics (Augmented Reality), as well as graphics augmented by reality (Augmented Virtuality). However, the taxonomy also includes modification of reality or virtuality or any combination of these. The modification is denoted by moving up the mediality axis. Further up this axis, for example, we can find mediated reality, mediated virtuality, or any combination of these. Further up and to the right we have virtual worlds that are responsive to a severely modified version of reality. (at right) Mediated reality generalizes the concepts of mixed reality, etc.. It includes the virtuality reality continuum (mixing) but also, in addition to additive effects, also includes multiplicative effects (modulation) of (sometimes deliberately) diminished reality. Moreover, it considers, more generally, that reality may be modified in various ways. The mediated reality framework describes devices that deliberately modify reality, as well as devices that accidentally modify it. More recently, the term augmented reality has been blurred a bit due to the increased interest of the general public in AR. Commonly known examples of AR are the yellow “first down” lines seen in television broadcasts of American football games using the 1st & Ten system, and the colored trail showing location and direction of the puck in TV broadcasts of ice hockey games.
The real-world elements are the football field and players, and the virtual element is the yellow line, which is drawn over the image by computers in real time. Similarly, rugby fields and cricket pitches are branded by their sponsors using Augmented Reality; giant logos are inserted onto the fields when viewed on television. In some cases, the modification of reality goes beyond mere augmentation. For example, advertisements may be blocked out (partially or wholly diminished) and replaced with different advertisements. Such replacement is an example of Mediated reality, a more general concept than AR. Television telecasts of swimming events also often have a virtual line which indicates the position of the current world record holder at that time. Another type of AR application uses projectors and screens to insert objects into the real environment, enhancing museum exhibitions for example. The difference to a simple TV screen for example, is that these objects are related to the environment of the screen or display, and that they often are interactive as well. Many first-person shooter video games simulate the viewpoint of someone using AR systems. In these games the AR can be used to give visual directions to a location, mark the direction and distance of another person who is not in line of sight, give information about equipment such as remaining bullets in a gun, and display a myriad of other images based on whatever the game designers intend. This is also called the head-up display. In some current applications like in cars or airplanes, this is usually a head-up display integrated into the windshield. The F-35 Lightning II has no Head-up display. This is because all targets are tracked by the aircraft’s situational awareness and the sensor fusion is presented in the pilot’s helmet mounted display.
The helmet mounted display provides an augmented reality system that allows the pilot to look through his own aircraft as if it wasn’t there. 1957-62: Morton Heilig, a cinematographer, creates and patents a simulator called Sensorama with visuals, sound, vibration, and smell. 1966: Ivan Sutherland invents the head-mounted display suggesting it was a window into a virtual world. 1975: Myron Krueger creates Videoplace that allows users to interact with virtual objects for the first time. 1989: Jaron Lanier coins the phrase Virtual Reality and creates the first commercial business around virtual worlds. 1992: Tom Caudell coins the phrase Augmented Reality while at Boeing helping workers assemble cables into aircraft. 1992: L.B. Rosenberg develops one of the first functioning AR systems, called VIRTUAL FIXTURES, at the U.S. Air Force Armstrong Labs, and demonstrates benefit on human performance.1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present first major paper on an AR system prototype, KARMA, at the Graphics Interface conference. 1999: Hirokazu Kato created ARToolKit at HITLab, where AR later is further developed by other HITLab scientists and it is demonstrated at SIGGRAPH that year. 2000: Bruce H. Thomas develops ARQuake, the first outdoor mobile AR game, and is demonstrated in the International Symposium on Wearable Computers. 2008: Wikitude AR Travel Guide launches on Oct. 20, 2008 with the G1 Android phone.  2009: Wikitude Drive – AR Navigation System launches on Oct. 28, 2009 for the Android platform.  2009: AR Toolkit is ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser. The main hardware components for augmented reality are: display, tracking, input devices, and computer. Combination of powerful CPU, camera, accelerometers, GPS and solid state compass are often present in modern smartphones, which make them prospective platforms for augmented reality. There are three major display techniques for Augmented Reality: Head Mounted Displays; Handheld Displays and Spatial Displays.
A Head Mounted Display (HMD) places images of both the physical world and registered virtual graphical objects over the user’s view of the world. The HMD’s are either optical see-through or video see-through in nature. An optical see-through display employs half-silver mirror technology to allow views of physical world to pass through the lens and graphical overlay information to be reflected into the user’s eyes. The HMD must be tracked with a six degree of freedom sensor. This tracking allows for the computing system to register the virtual information to the physical world. The main advantage of HMD AR is the immersive experience for the user. The graphical information is slaved to the view of the user. The most common products employed are as follows: MicroVision Nomad, Sony Glasstron, and I/O Displays. Handheld Augment Reality employs a small computing device with a display that fits in a user’s hand. All handheld AR solutions to date have employed video see-through techniques to overlay the graphical information to the physical world. Initially handheld AR employed sensors such as digital compasses and GPS units for its six degree of freedom tracking sensors. This moved onto the use of fiducially marker systems such as the ARToolKit for tracking. Today vision systems such as SLAM or PTAM are being employed for tracking. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR is the portable nature of handheld devices and ubiquitous nature of camera phones. Instead of the user wearing or carrying the display such as with head mounted displays or handheld devices; Spatial Augmented Reality (SAR) makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Because the displays are not associated with each user, SAR scales naturally up to groups of users, thus allowing for collocated collaboration between users. SAR has several advantages over traditional head mounted displays and handheld devices.
The user is not required to carry equipment or wear the display over their eyes. This makes spatial AR a good candidate for collaborative work, as the users can see each other’s faces. A system can be used by multiple people at the same time without each having to wear a head mounted display. Spatial AR does not suffer from the limited display resolution of current head mounted displays and portable devices. A projector based display system can simply incorporate more projectors to expand the display area. Where portable devices have a small window into the world for drawing, a SAR system can display on any number of surfaces of an indoor setting at once. The tangible nature of SAR makes this an ideal technology to support design, as SAR supports both a graphical visualization and passive haptic sensation for the end users. People are able to touch physical objects, and it is this process that provides the passive haptic sensation. Modern mobile augmented reality systems use one or more of the following tracking technologies: digital cameras and/or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID, wireless sensors. Each of these technologies have different levels of accuracy and precision. Most important is the tracking of the pose and position of the user’s head for the augmentation of the user’s view. The user’s hand(s) can tracked or a handheld input device could be tracked to provide a 6DOF interaction technique. Stationary systems can employ 6DOF track systems such as Polhemus, ViCON, A.R.T, or Ascension.This is a current open research question. Some systems, such as the Tinmith system, employ pinch glove techniques. Another common technique is a wand with a button on it. In case of Smartphone, phone itself could be used as 3D pointing device, with 3D position of the phone restored from the camera images. Camera based systems require powerful CPU and considerable amount of RAM for processing camera images. Wearable computing systems employ a laptop in a backpack configuration. For stationary systems a traditional workstation with a powerful graphics card. Sound processing hardware could be included in augmented reality systems. For consistent merging real-world images from camera and virtual 3D images, virtual images should be attached to real-world locations in visually realistic way.
That means a real world coordinate system, independent from the camera, should be restored from camera images. That process is called Image registration and is part of Azuma’s definition of Augmented Reality. Augmented reality image registration uses different methods of computer vision, mostly related to video tracking. Many computer vision methods of augmented reality are inherited form similar visual odometry methods. Usually those methods consist of two parts. First interest points, or fiduciary markers, or optical flow detected in the camera images. First stage can use Feature detection methods like Corner detection, Blob detection, Edge detection or thresholding and/or other image processing methods. In the second stage, a real world coordinate system is restored from the data obtained in the first stage. Some methods assume objects with known 3D geometry(or fiduciary markers) present in the scene and make use of those data. In some of those cases all of the scene 3D structure should be pre calculated beforehand. If not all of the scene is known beforehand SLAM technique could be used for mapping fiduciary markers/3D models relative positions. If no assumption about 3D geometry of the scene made structure from motion methods are used.
Methods used in the second stage include projective geometry, bundle adjustment, rotation representation with exponential map, kalman and particle filters. Advertising: Marketers started to use AR to promote products via interactive AR applications. For example, at the 2008 LA Auto Show, Nissan unveiled the concept vehicle Cube and presented visitors with a brochure which, when held against a webcam, showed several versions of the vehicle. In August 2009, Best Buy ran a circular with an augmented reality code that allowed users with a webcam to interact with the product in 3D. Support with complex tasks: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system. AR can include images of hidden objects, which can be particularly effective for medical diagnostics or surgery. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound or open NMR devices. A doctor could observe the fetus inside the mother’s womb. Navigation devices: AR can augment the effectiveness of navigation devices for a variety of applications.
For example, building navigation can be enhanced for the purpose of maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can be used to provide navigation hints and traffic information. These types of displays can be useful for airplane pilots, too. Head-up displays are currently used in fighter jets as one of the first AR applications. These include full interactivity, including eye pointing. Industrial Applications: AR can be used to compare the data of digital mock-ups with physical mock-ups for efficiently finding discrepancy’s between the two sources. It can further be employed to safeguard digital data in combination with existing real prototypes, and thus save or minimize the building of real prototypes and improve the quality of the final product. Military and emergency services: AR can be applied to military and emergency services as wearable systems to provide information such as instructions, maps, enemy locations, and fire cells. Prospecting: In the fields of hydrology, ecology, and geology, AR can be used to display an interactive analysis of terrain characteristics. Users could use, and collaboratively modify and analyze, interactive three-dimensional maps. Art: AR can be incorporated into artistic applications that allow artists to create art in real time over reality such as painting, drawing, modeling, etc. One such example of this phenomenon is called Eyewriter that was developed in 2009 by Zachary Lieberman and a group formed by members of Free Art and Technology (FAT), OpenFrameworks and the Graffiti Research Lab to help a graffiti artist, who became paralyzed, draw again. Architecture: AR can be employed to simulate planned construction projects. Sightseeing: Models may be created to include labels or text related to the objects/places visited. With AR, users can rebuild ruins, buildings, or even landscapes as they previously existed. Collaboration: AR can help facilitate collaboration among distributed team members via conferences with real and virtual participants. The Hand of God is a good example of a collaboration system.
AR can be used in the fields of entertainment and education to create virtual objects in museums and exhibitions, theme park attractions (such as Cadbury World), and games (such as ARQuake and The Eye of Judgment). Also see Mixed reality. Music: Pop group Duran Duran included interactive AR projections into their stage show during their 2000 Pop Trash concert tour. Sydney band Lost Valentinos launched the world’s first interactive AR music video on 16 October 2009, where users could print out 5 markers representing a pre-recorded performance from each band member which they could interact with live and in real-time via their computer webcam and record as their own unique music video clips to share via YouTube. It is important to note that augmented reality is a costly development in technology. Because of this, the future of AR is dependent on whether or not those costs can be reduced in some way. If AR technology becomes affordable, it could be very widespread but for now major industries are the sole buyers that have the opportunity to utilize this resource. Expanding a PC screen into the real environment: program windows and icons appear as virtual devices in real space and are eye or gesture operated, by gazing or pointing.
A single personal display (glasses) could concurrently simulate a hundred conventional PC screens or application windows all around a user. Virtual devices of all kinds, e.g. replacement of traditional screens, control panels, and entirely new applications impossible in “real” hardware, like 3D objects interactively changing their shape and appearance based on the current task or need. Enhanced media applications, like pseudo holographic virtual screens, virtual surround cinema, virtual ‘holodecks’ (allowing computer-generated imagery to interact with live entertainers and audience) Virtual conferences in “holodeck” style. Replacement of cell phone and car navigator screens: eye-dialing, insertion of information directly into the environment, e.g. guiding lines directly on the road, as well as enhancements like “X-ray”-views Virtual plants, wallpapers, panoramic views, artwork, decorations, illumination etc., enhancing everyday life. For example, a virtual window could be displayed on a regular wall showing a live feed of a camera placed on the exterior of the building, thus allowing the user to effectually toggle a wall’s transparency.
With AR systems getting into mass market, we may see virtual window dressings, posters, traffic signs, Christmas decorations, advertisement towers and more. These may be fully interactive even at a distance, by eye pointing for example. Virtual gadgetry becomes possible. Any physical device currently produced to assist in data-oriented tasks (such as the clock, radio, PC, arrival/departure board at an airport, stock ticker, PDA, PMP, informational posters/fliers/billboards, in-car navigation systems, etc.) could be replaced by virtual devices that cost nothing to produce aside from the cost of writing the software. Examples might be a virtual wall clock, a to-do list for the day docked by your bed for you to look at first thing in the morning, etc. Subscribable group-specific AR feeds. For example, a manager on a construction site could create and dock instructions including diagrams in specific locations on the site. The workers could refer to this feed of AR items as they work. Another example could be patrons at a public event subscribing to a feed of direction and information oriented AR items. AR systems can help the visually impaired navigate in a much better manner (combined with a text-to-speech software). Augmented reality is one of the newest innovations in the electronics industry. It superimposes graphics, audio and other sense enhancements from computer screens onto real time environments. Augmented reality goes far beyond the static graphics technology of television where the graphics imposed do not change with the perspective. Augmented reality systems superimpose graphics for every perspective and adjust to every movement of the user’s head and eyes.
Development of the needed technology for augmented reality systems, however, is still underway within the laboratories of both universities and high tech companies. It is forecasted that by the end of this decade, the first mass-produced augmented reality systems will hit the market. The three basic components of an augmented reality system are the head-mounted display, tracking system and mobile computer for the hardware. The goal of this new technology is to merge these three components into a highly portable unit much like a combination of a high tech walkman and an ordinary pair or eyeglasses. The head-mounted display used in augmented reality systems will enable the user to view superimposed graphics and text created by the system. As of today, the technology nearest to augmented reality head mounts is one that is being used in virtual reality applications. There are two basic head mount design concepts that are being researched for augmented reality systems and these are the video see-through systems and optical see-through systems. The video see-through systems block out the user’s view of the outside environment and play the image real time through a camera mounted on the head gear. The main problem with this type of system is the delay in image adjustment whenever the user moves his head. Optical see-through systems, on the other hand, make use of technology that “paints” the images directly onto the user’s retina through rapid movement of the light source. Though this system has its drawbacks, particularly its high price, researchers are confident that this system will be a lot more portable and less inconspicuous for future augmented reality systems. Another component of an augmented reality system is its tracking and orientation system. This system pinpoints the user’s location in reference to his surroundings and additionally tracks the user’s eye and head movements.
The complicated procedure of tracking overall location, user movement and adjusting the displayed graphics needed are some of the major hurdles in developing this technology. So far, the best systems developed still presents a lag or a delay between the user’s movement and the display of the image. Last but not the least; augmented reality systems will need highly mobile computers. As of now, available mobile computers that can be used for this new technology are still not sufficiently powerful to create the needed stereo 3-D graphics. Graphics processing units like the NVidia GPU by Toshiba and ATI mobility 128 16MB-graphics chips are however being integrated into laptops to merge the current computer technology to augmented reality systems. The potential uses of augmented reality systems in everyday living and in various fields are many. Once available in the market, augmented reality systems will change the way people see and learn from their surroundings. Following are several applications for augmented reality systems. Augmented reality systems can be used to enhance gaming and entertainment. RPG games in the future can be integrated with augmented reality systems to give the user real environments as backdrops for his game and to make the user’s senses perceive that he truly is one of the characters in the game. Sports fans will have access to up to date game information and enhancd sports viewing at home. Augmented reality systems in combination with other technologies such as WiFi could also be used to provide instant information to its users. For educational purposes, augmented reality systems can be used to view a panoramic recreation of a historical event superimposed on its real-time background. Students could use this system to have a deeper understanding on things like the formation of clouds, the structure of the universe and the galaxy, etc. through realistic and easily understandable augmented reality systems simulations.
The military, particularly the Office of Naval Research and Defence Advanced Research Projects Agency or DARPA, are some of the original pioneers of augmented reality systems. One of the main uses of augmented reality systems to the military is providing field soldiers crucial information about their surroundings as well as friendly troops and enemy movements in their particular area. Augmented reality systems will also play a big role in law enforcing and intelligence agencies. This system will enable police officers to have a complete and detailed view and information about a crime scene, a patrol area, or a suspect line up. Medically, augmented reality systems could be used to give the surgeon a better sensory perception of the patient’s body during an operation. This will result in less risky and more efficient surgical operations. The system could also be used in conjunction with other medical equipments such as an x-ray machine or an MRI to instantly give the doctors the information they need to make a medical diagnosis or decision. The building and construction field will benefit from the easier project management that augmented reality systems will bring. Markers can be placed or attached to a particular object a person is currently working on so that project and site managers can monitor work in progress. In the petroleum and mining industry, it will enable decision makers to make timely decisions. The management can decide about how ore will be mined by merely looking at the superimposition of field data fed by the geological survey team through the augmented reality systems. Augmented reality systems can be used in almost any field or industry. The novelty of instant information coupled with enhanced perception will ensure that augmented reality systems will play a big role in how people live in the future.