Page 42 - Spring2020

Music Beyond Sound
The mechanism that my group has proposed as underpinning the visual contributions to melodic pitch perception involves feedforward and feedback connections along a dorsal stream connecting the sensory and motor areas. Feedforward con- nections from sensory areas to motor areas provide input that enable an internal motor simulation of what is being perceived. Motor areas feedback information about biological movement that can be compared with the incoming sensory information (Kilner et al., 2007). In the case of individuals who are hard of hearing, there may also be an additional contribution that stems from visual activation of the auditory cortex, particularly in the belt areas (Finney, 2001; Good et al., 2014). Research in animal models suggests that the belt areas of the auditory cortex undergo profound plastic changes following a period of auditory deprivation, which leads in some cases to enhanced visual processing.
The effects of vision are also invoked in the perception of musi- cal timbre. In a musical extension of the well-known “McGurk effect,” Saldaña and Rosenblum (1993) presented participants with audiovisual recordings of cello tones where bowing and plucking information was manipulated across channels. So, for example, observers were presented with an audiovisual record- ing in which the audio channel consisted of plucking and the visual channel consisted of bowing. The results revealed that plucking sounds were more likely to be heard as bowing when accompanied by the sight of bowing. These results were inter- preted as evidence consistent with an automatic internal motor simulation. The authors further presume that the simulation may be driven by auditory or visual information.
A number of visual contributions to rhythm perception have been established (Schutz, 2008). For example, Rosenblum and Fowler (1991) recorded handclaps of varying intensity. Participants were presented with audiovisual pairings of the handclaps that were congruent (from recordings of simi- lar intensity) or incongruent (from recordings of different intensity). Results revealed that the visual information had a systematic influence on loudness judgments despite the instruction to focus on auditory information only. Visual information can also influence the perceived duration of a performed note. To study this phenomenon, Schutz and Lip- scomb (2007) utilized audiovisual recordings of marimba notes performed using “long” (i.e., exaggerated) and “short” gestures. Much like Rosenblum and Fowler (1991), visual channels were recombined to form congruent and incongruent pairings. In this example, the auditory channel had no effect on judgments even though the participants were asked to focus their judg-
ments on the auditory channel. However, the visual channel has a consistent effect on the perceived duration such that long gestures lengthened the perceived duration of notes and short gestures shortened them. In a follow-up study, the visual chan- nel was replaced with a point-light rendering of the performer’s movement. Results showed that the visual effect on judgments remained, which suggests that the visual effect is based on the dynamics of visual movement (Schutz and Kubovy, 2009).
Some evidence suggests that the ability to synchronize with a rhythm (i.e., tap along or dance in time) depends on the nature of the visual stimulus. Although discrete visual stimuli (e.g., flashes) have been found to be inferior to auditory tones (Patel et al., 2005), continuous visual stimuli (e.g., a bouncing ball) lead to near comparable synchronization performance (Iversen et al., 2015). Neuroimaging studies have demonstrated that continuous visual stimuli give rise to greater activation of the putamen (a brain area that is strongly implicated in beat perception) than do discrete visual stimuli, approaching levels of activation obtained with auditory beeps (Grahn, 2012). This finding suggests that the ability to synchronize to metrical structure is not simply contingent on the channel of sensory input but also on the nature of stimulus presentation. Although discrete events are optimal with auditory stimuli, continuous events lead to better outcomes with visual stimuli. Some evi- dence suggests that deaf individuals possess some advantage in tracking visual rhythms (Iversen et al., 2015). Referring back to Figure 2, the strength of direct visual input to auditory-motor pathways is likely enhanced in deaf individuals.
Getting the Balance Right
Both passive and active head movements are capable of stimu- lating the vestibular system that is involved with the sense of balance (Cullen and Roy, 2004). With the possible exception of classical music performances, it is common to see people moving their heads while listening to music. As such, it would seem that vestibular stimulation is commonplace during music listening (see Figure 6). Moreover, given that the vestibular cortex is extensively connected with other sensory cortices, it stands to reason that there are ample opportunities for multi- sensory integration in music that involve the vestibular system.
Phillips-Silver and Trainor (2005) assessed the contribution of the vestibular system to multisensory rhythm using an ambiguous auditory rhythm (see tinyurl.com/wwry2qu for an auditory example). These rhythms can be heard in duple or triple form, that is to say, recurring patterns of two beats (as in a march) or three beats (as in a waltz). The rhythms
42 | Acoustics Today | Spring 2020