Ron Azuma and HRL Laboratories is doing some really nice work in this area, using video tracking with tagged objects. They use video see-through (in which the user is looking at a video image of the scene in front of them instead of the real scene), over which information about tagged objects is placed. Text can be hard to read at video resolution, but Azuma reports that SRI is working on a 2k x 2k resolution CMOS video camera. [You could do it with HD, but it's too expensive right now. I predict, however, that cheap, hi-res cameras, perhaps Firewire, will become common soon.]

Another problem with video see-through is that redraw can lag when the user moves his head (on which the camera is mounted) too quickly. To address this, HRL is using accelerometers on the head-mounted display for motion compensation, a solution which works well -- overlaid information stays with a real-world object as the head moves. Other sensors can include optical, magnetic, and compasses, but since these are also subject to interference and drift errors, a hybrid system works best. The Kalman Filter is used to parse all the incoming signals. (See also a SIGGRAPH 1997 paper on SCAAT, which describes how to read a single coordinate at a time.)

Object tagging is a particularly interesting area. Most people are using video tracking and pattern recognition to read visual icons printed onto various surfaces. This becomes an interesting semiotics issue, as the real world becomes tagged with symbols readable only to computers, though some people are using recognizable symbols such as Kanjii characters. The angle and rotation of icons can be tracked by using color to identify the edges.

Azuma demonstrated a small, handheld device with alphanumeric LCD display, to capture information from objects. This led to some interesting variations, such a system with headmounted display and projections, in which you could use the device to grab a virtual object off of the projected image, and place it in the real world (as you see it through your HMD).

The output need not be a display -- sound works quite well, especially when spatialized. Audio coming into the computer can also be tracked spatially. Augmented reality can be tangible and wireless. Really interesting things will happen as displays become less cumbersome and high-res, moving graphics can be superimposed flawlessly onto a scene, making the most of natural features. For now, there are seams -- spatial, temporal, functional, but these are being addressed. But reality can be augmented, mixed or virtual, and so can virtuality. Video tracking makes AR closer to VR, as a user could transmit the captured video feed to somewhere else, merging task space and communication space. AR shows promise for architects to envision new buildings in their surroundings. [Does it make a difference if buildings increasingly look comptuer-generated?]

Also demonstrated was videoconferencing in which live, chromakeyed video images of participants are placed in AR space or mapped onto 3D objects in VR. Sketching was also interesting (see graffitti). But collaboration is difficult when two participants' eyes are covered by goggles; new eyeglasses displays are helping. And aren't eyeglasses a technology for augmenting reality anyway?

I said earlier that interface design was increasingly important. Right now, some of the best AR interfaces have been developed for video games, since first-person shooters, in particular, have been working on this for several years. A variation on this is the University of South Australia's AR version of Quake, in which monsters are superimposed onto the real world. A general benefit of the SIGGRAPH conference is the symbiosis of university researchers, game developers, and effects artists.

Dieter Schmalstieg of Technischen Universitat Wien echoed the need for multimodal or heterogeneous interfaces. He compared AR to looking through lenses, and focused on optical see-through, in which only data is superimposed on a transparent display. He also brought up the issue of privacy management in a multi-user AR environment -- the possibility that all users may not need to see all data.

Tobias Hollerer of Columbia addressed mobile AR, as might be used by a journalist to annotate a scene or colllaborate with another person. What Hollerer described was physical hypermedia, a variant of the physical computing done downtown at NYU. John Pavlik at Columbia has been doing much work in this area. Hollerer detailed the march of shrinking components, mentioning the Minolta Forgettable Display, Kaiser Electro-Optical, and modifying a sub-notebook computer for mobile AR use by removing the screen (Toshiba and Dell were suggested.) In terms of interface design, Hollerer and his group have found that it can be jarring to read information that keeps moving in an HMD, so they have explored keeping some information "world stabilized," and using a line to link the world-stabilized and screen-stabilized data. A variation is body-stabilized data, which aligns with the orientation of the user's body but not head.

Sensor technologies mentioned included the MIT "locust swarm," ultrasound work being done by AT&T Cambridge, GVU. Hollerer has been working with GPS, and says that its resolution (normally 10 to 30 meters) is improved by a technique called RTK differential GPS, or by using "pseudolites" -- terrestrial ground transceivers instead of satellites. But GPS still requires good line-of-sight and is subject to interference. Better methods are dead reckoning or inertial systems

Are interfaces are intruding on the real world, or is the real world increasingly being captured into the digital world?

Kevin Walker