Human Systems Integration Division @ NASA Ames

While the use of hand tools and other everyday manually controlled devices is naturally accompanied by multisensory feedback, the deployment of fully multimodal virtual interfaces requires that haptic, acoustic, and visual cues be synthesised. The complexity and character of this synthesis will depend on a thorough understanding of the multimodal perceptual experience, including the interrelations between the individual sensory channels during manual interaction. In this study seventy participants were asked to rank the manual operation of ten electromechanical switches according to preference. The participants were randomly assigned in groups of ten to one of seven sensory presentation conditions. These conditions comprised six bimodal and unimodal sensory combinations created by selectively restricting the flow of haptic, auditory, and visual information, plus one condition in which full sensory information was available. A principal components analysis on the obtained ranking data indicated that the sensory conditions with unimpeded haptic information were clearly distinct from those in which the haptic cues were impeded. The analysis also showed that, for switch use, the unimodal haptic condition most closely approached the condition with combined haptic, auditory, and visual feedback, compared with all of the conditions where haptic feedback was restricted.