Page 41 - Spring2020

gertip (Brochard et al., 2008), toe (Müller et al., 2008), and back (Ammirante et al., 2016). In the case of a simple beat like that produced by a metronome (isochronous), synchronization is equivalent for vibrotactile and auditory presentations, at least under some presentation conditions. For example, Müller et al. (2008) found that the ability to tap to the beat was comparable when the vibrotactile stimulation was applied to the fingertip, but performance dropped when the stimulation was applied to the toe. Ammirante et al. (2016) found that synchronization ability was equivalent when the input stimulus was sound or vibrotactile stimulation on the back, but only if the area of stimu- lation on the back was large. In the same study, synchronization ability was found to be superior for auditory stimulation when the rhythms were more complex than a metronome. Again, this effect may be related to the vast experience amassed with moving to music presented as sound. Along these same lines, Lauzon et al. (2020) found that the ability to detect asynchronies in a rhythm were superior when the rhythm was presented by auditory compared with vibrotactile stimulation.
Vibrotactile stimuli have been shown to activate belt areas of the auditory cortex bilaterally (Schürmann et al., 2006). The extent of auditory activations observed in deaf participants is more widespread than that observed in normal-hearing par- ticipants (Auer et al., 2007), likely due to rewiring in the brain that follows a period of sensory deprivation. One question resulting from this work is whether activation of auditory areas by vibrotactile stimuli is direct or whether the auditory activa- tion arises indirectly as a result of projections from touch areas.
Using magnetoencephalography, Caetano and Jousmäki (2006) were able to track the time course of activations cor- responding to different sensory cortices. Normal-hearing participants were presented with vibrotactile stimulation at 200 Hz delivered to the fingertips. An initial response was observed in the primary touch (somatosensory) cortex, peaking at around 60 ms poststimulus, followed by tran- sient responses in auditory cortex between 100 and 200 ms. Finally, a sustained response was observed in the auditory cortex between 200 and 700 ms. These findings suggest that in normal-hearing listeners, at least, auditory representations of vibrotactile stimuli are made possible by a causal process- ing chain that starts in the somatosensory cortex that then feeds forward into the auditory cortex.
Facing the Music
A melodic interval is produced when two notes are played in succession. Larger melodic intervals involve a greater pitch
Figure 5. Movements of the head, eyebrows, and mouth in a vocal performance provide reliable information about melodic interval size.
distance between notes. Several studies have shown that observers can detect the size of a sung melodic interval on the basis of visual observation of the performer’s head and face. When silent videos of sung melodic intervals are presented to observers, they are able to accurately scale their relative size (Thompson and Russo, 2007). This ability does not appear to require music or vocal training, which argues against an explanation based on long-term memory and further sug- gests that some aspects of the visual information provide reliable cues for judging interval size. Video-based tracking has supported this interpretation, revealing that larger inter- vals tend to possess more head movement, eyebrow raising, and mouth opening. The influence of visual information on the perception of size in sung melodic intervals persists even when videos are converted into point-light displays in which the dynamic information is retained through a matrix of dots while eliminating static visual cues (Abel et al., 2016).
Sight can influence the perceived size of sung melodic intervals even when the sound is present (Thompson et al., 2010). The mouth area may be particularly important in this visual effect (see Figure 5). In one study involving audiovisual presenta- tions of sung intervals, the signal-to-noise ratio (SNR) in the audio channel was manipulated across conditions. As the SNR of the sung melodic intervals decreased, the extent to which participants directed their gaze toward the mouth increased (Russo et al., 2011). However, the visual influence on audi- tory judgments has been found to be reduced for participants with a young onset of musical training (Abel et al., 2016). One interpretation of this latter finding is that early-trained musi- cians possess a stronger audiomotor representation of sung melodic intervals. This enhancement in motor representation may allow these early-trained musicians to rely less heavily on sight when presented with audiovisual input.
Spring 2020 | Acoustics Today | 41