The HIT Lab at U of Wash. does interesting work, and Richard May mentioned this virtual dig project for the Seattle Art Museum. Similar to something we did at AMNH in 2000 (http://research.amnh.org/exhibitions/dinos/dig3.jpg) in which you can have multiple people using multiple hands (in Seattle's case plastic tools) to "dig" on a rear-projection screen.

"The interaction between display characteristics and the information processing characteristics of the user's perceptual, motor, and cognitive processes will largely determine interface performance." The computer metaphor of the mind views it as a single processor which operates on a variety of sensory inputs, whereas functional neurophysiology and sensory psychophysics research indicate that processing is divided into the aforementioned modules. This can be applied to interface design in distinguishing the "where" (perception) from the "how" (action), and recognizing that different brain areas process information we merely need to recognize, and that which requires us to take action.

Referring to the latter, it has been demonstrated that symbolic representations combined with pointers (such as an arrow) can greatly aid in identifying information which requires our action. (In the HCI community these are referred to as affordances -- cues which afford action.) Research has shown that the brain can generally only process no more than four of these attentional tokens simultaneously.

Also applicable, if somewhat obvious, is that multiple sensory modes can reinforce eachother. This is particularly true for complex phenomena, where auditory cues, for example, might supplement visual or haptic ones. Conversely, of course, conflicting sensory cues can cause confusion. Ronald Resnick described the phenomenon of "change blindness," "the inability to see large changes that are made at the same time as a disturbance elsewhere in the display.... More precisely, our visual experience is based on coherent representations that are formed whenever attention is allocated to them, and which dissolve as soon as attention is withdrawn.... In light of this, it is suggested that interface design should rely not only on knowledge of the attentional system (which provides visual experience), but also of the nonattentional systems that guide it."

Karon MacLean described "physical interaction design" -- the use of haptic devices in multimodal interfaces. She divides haptics into two types of sensory inputs -- taction (heat, vibration, pressure, slip, pain), and kinethesia (limb position, motion, force). Haptic "force-feedback" devices are usually targetted at one or the other, and especially the latter. There are actuators which deliver force, torque, motion, temperature, pressure, even electric shock! There are motors, pneumatics, hydraulics, brakes, piezos, even shape memory alloys. Of course, there is the ubiquitous pager vibrating motor. Devices such as these provide a two-way exchange between the user and computer -- they act as control and display channel at the same time. Up to now, they have been used primarily for medical applications such as surgery simulation, but are starting to move into the military and consumer sectors. The first commercial device of this sort was the Phantom, a pen-like devices attached to actuators, which inhibit motion when the user bumps up against some virtual obstruction. This is an example of a "grounded" haptic device -- it is solidly connected to the world (or exhibit); others are "ungrounded," attached to the body or which follow you around.

Note that tactile stimuli are sensed in the 10 to 10,000 Hz range, putting them within range of audible range vibrations. This makes sense, since sound is detected physically by vibrations of tiny hairs in the cochlea. Kinesthetic forces are in the 20 to 30 Hz range. A "control bandwidth" of 5 to 10 Hz indicates how fast we can move our limbs or digits -- much lower frequency than the rate of motion we can perceive.

MacLean divides haptic devices into two general categories: "buttons" (things which provide discrete control and information), and "handles" (those which provide continuous monitoring and control). Haptic design must take into account the fact that sensory receptors can become adapted to a continuous stimulus (you don't notice a loud, steady noise after a while). Also, note changes in spatiotemporal resolution: the location of a particular stimulus can become confusing if there is crosstalk or overlap; and stimuli need to be spaced more than 5 milliseconds apart in order to be perceived as separate. She also points out that haptic interfaces can often fail if users are inhibited from touching something if they perceive it to be dirty, painful, forbidden, or intimate.

As above, supplementing haptic interfaces with other sensory inputs can be effective. Research has shown, for example, that partially deaf people tend to hear better in light than dark; and thresholds for touch are lowered by exposure to weak sounds but raised when the sound is intensified. However, when one of two conflicting stimuli dominates, it is said to "capture" the other, and vision and audition tend to capture haptic perception in general, but this depends on the task and attentional requirements. A further complication is that vision and audition are refreshed at high frequencies (30 to 100 Hz) while the fastest haptic loop is only 0.5 to 10 Hz.

More on haptics:
http://www.VR2002.org/
http://www.haptics-e.org/
http://www.cs.ubc.ca/~cs532/references/readings.html
http://www.cs.ubc.ca/spin/publications/index.html
http://haptic.mech.nwu.edu/
http://www.roblesdelatorre.com/gabriel/hapticsl
http://www.eurohaptics.org/

Commercial products:

Sidney Fels talked about the physiology behind our different senses. He explained that the fovea in each of your eyes is composed of cones, which have high resolution but fairly low sensitivity; the periphery is mostly rods, which are more sensitive but have lower resolution. It takes 8 to 10 Hz to give a sensation of motion. In order to provide full resolution to the eye, you need 8000 x 8000 pixels. For a sense of immersion, you need to provide a field of view of 60 to 100 degrees (hence domes and wraparound screens). The eye detects flickering below 50 to 60 Hz (though this changes with age). Broadcasting uses 50 Hz (PAL) or 60 Hz (NTSC) while computer monitors go from 60 up to 120 or so. We are more sensitive to changes in the intensity of a stimulus than the intensity itself.

Depth is determined not only by the binocular arrangement of our two eyes, but by many other factors such as overlap, relative size, atmosphere, perspective, parallel line convergence, etc. Each of these can be exploited separately to give, or contribute to, an illusion of depth. There is much work going on in autostereoscopic displays (for which you don't need glasses to see 3D images). For example, see

http://www.mrl.nyu.edu/projects/autostereo/
http://www.inition.co.uk/inition/product_stereovis_stereographics_synthagram.htm
- DTI - Dimension Technologies Inc. (parallax barrier)
- Philips 3D-LCD (lenticular lenses)
- Richmond Holographic Studios Ltd. (RHS) (holographic optical elements - HOEs)
- Sanyo 3D Screen (image splitter with head tracking)
- VISUREAL Displaysysteme GmbH (Holotron)
- RealityVision (HOE)
- Technische Uni Dresden - D4D (image splitting with head tracking)
- 3D EXPERIENCE LtdSpexfree 3D Monitor (looks like image splitting)
- Dimensional Media Associates: high definition volumetric display (HDVD) - two concave mirrors + audio control optics
- CRL (Kakeya et. Al.) use Fresnel lens plus comp. Cont. polarizing filters.

"One thing to remember, illusions are your friends. When you encounter an illusion as a designer, you should be thinking, 'how can I take advantage of this?' You want to think this since illusions are windows into how the brain works and the assumptions it makes about the real world. The right illusion coupled with the appropriate cues from the other sensory channels can give a powerful sense of immersion."

Sound travels 350m/s and you can perceive sounds in the range of 16 to 20,000 Hz when a pressure waveform stimulates the 23,500 hair cells in your ears. Perceived loudness is logarithmic, and perceived sound is highly dependent on the environment -- if the reverberation is not correct, it doesn't sound right. A delay of 30 to 50 milliseconds is perceived as echo. Your ears localize sounds by sensing the differences in intensity and time between the two ears. Intensity changes are perceived at higher frequencies (above 1kHz) and time differences are perceived at lower than 1kHz. The ear shape also acts as a directional filter.

There are many potential input channels that are largely unexplored: breathing, facial control, gait, gesture, galvanic skin response, heartrate, brain activity.

Interesting case study was the . The MIT Media Lab sought to engage multiple people (kids) to interact with eachother as well as projected virtual stories and characters, in a controlled physical space, using non-encumbering sensing technologies. It was a fully-automated, 10- to 12-minute interactive experience for up to 4 people.

They found that group interaction dictated a faster pace to the experience, and that cause and effect relationships were often not clear to the users. This is especially true if there is a perceptual lag, something we have learned from the neurophysiological research above. They also found that visual cues (FINSTs or affordances as noted above) would have been helpful. But they did find that a linear story could still be interactive, particularly if it involved group activity.

They ran into technological hurdles: The camera lenses weren't wide enough to track everyone, everywhere. They found that when you put a lot of projection screens into a room, they act as very large light sources. This was particularly problematic for visual tracking, as the changing lighting conditions and shadows confused the software. They found that they could not do speech recognition in a loud environment in which multiple people might be talking (or yelling). They also found that they could control computer-human interaction somewhat, but not human-human interaction.

A big problem with any such project is how to determine context. Sensors can deliver data, but data depend on the context of the action.

Claudio Pinahez proposed a process for user-centered design:

A variant is using multiple projectors to project onto an irregular space, with software that blends the geometry and luminance to create an undistorted image. This views the projector as equivalent to the camera. The theory and math are spelled out in great detail in the course notes above.