Proceedings of Meetings on Acoustics

Transcription

1 Proceedings of Meetings on Acoustics Volume 19, ICA 2013 Montreal Montreal, Canada 2-7 June 2013 Speech Communication Session 4pSCb: Production and Perception I: Beyond the Speech Segment (Poster Session) 4pSCb34. Effects of musical experience on perception of audiovisual synchrony for speech and music Dawn Behne*, Magnus Alm, Aleksander Berg, Thomas Engell, Camilla Foyn, Canutte Johnsen, Thulasy Srigaran and Ane E. Torsdottir *Corresponding author's address: Psychology, NTNU, Dragvoll A, Trondheim, NO7491, Sør-Trondelag, Norway, Perception of audiovisual synchrony relies on matching temporal attributes across sensory modalities. To investigate the influence of experience on cross-modal temporal integration, the effect of musical experience on the perception of audiovisual synchrony was studied with speech and music stimuli. Nine musicians and nine non-musicians meeting strict group criteria provided simultaneity judgments to audiovisual /ba/ and guitar-strum stimuli, each with 23 levels of audiovisual alignment. Although results for the speech and music stimuli differed, the two groups did not differ in their responses to the two types of stimuli. Consistent with previous research, responses from both groups show less temporal sensitivity to stimuli with video-lead than audio-lead. No significant between-group difference was found for video-lead thresholds. However, both for the speech and music stimuli, musicians had an audio-lead threshold significantly closer to the point of physical synchrony than non-musicians, indicating the musicians greater acuity for audiovisual temporal coherence. Overall this leads to a non-significant tendency for a narrower window of synchrony for musicians than non-musicians. Findings are consistent with predictions that cross-modal temporal experience increases threshold acuity for audio-lead, but not for video-lead, and also support theories suggesting greater efficiency with relevant experience. Published by the Acoustical Society of America through the American Institute of Physics 2013 Acoustical Society of America [DOI: / ] Received 28 Jan 2013; published 2 Jun 2013 Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 1

2 INTRODUCTION Natural speech perception makes use of multisensory information. The unity assumption suggests that when that information reaches the different senses, the more properties they have in common, such as occurring close in time, the more likely the brain is assumed to treat them as having a single source (e.g., Welch and Warren, 1980). Precision in relating audio and visual information to a mutual source increases neural processing time (van Wassenhove, Grant, and Poeppel, 2005) and gives a perceptual benefit (e.g., Grant and Seitz, 1998; Sumby and Pollack, 1954). The window of perceived subjective simultaneity for audiovisual speech is roughly a few hundred milliseconds (e.g., Conrey and Pisoni, 2006; Grant et al., 2004; Hay-McCutcheon, Pisoni and Hunt, 2009; McGrath and Summerfield, 1985). However, the relative alignment of the audio and video signals has differing effects: to be perceived as asynchronous, a greater physical synchrony of video preceding audio is needed compared to audio preceding video, where a smaller audiovisual misalignment is needed for the asynchrony to be perceived (e.g., Conrey and Pisoni, 2006; Hay-McCutcheon et al., 2009). The asymmetry of subjective perception around the point of audiovisual synchrony has generally been ascribed to perceptual accommodation to differences in the propagation speeds of sound and light and their corresponding neural processing times for the different senses (Keetels and Vroomen, 2012). In short, these taken together, perception of visual-lead asynchronies will be most affected by the distance between the audiovisual event and the perceiver, and perception of the audio-lead asynchronies are primarily affected by neural processing. Furthermore, since articulatory movement precedes the speech signal (Smeele, Sittig, and Heuven, 1994), the visual information may provide the perceiver with a predictor for the auditory signal (Grant et al., 2004). From a statistical learning perspective, this may decrease the likelihood of experiencing audio information preceding visual cues and develop a perceptual limitation for misalignments when the audio signal precedes the video, but not the reverse. If so, this suggests that experience may influence the width and relative asymmetry associated with the window of perceived simultaneity, in particular by increasing sensitivity for an audio signal preceding a video signal. Previous research has shown individual differences in the window of perceived simultaneity in audiovisual speech perception (e.g., Grant and Seitz, 1998) and that the width of this window can have lasting effects from training (e.g., Powers, Hillock and Wallace, 2009). Summerfield (1992) recounts an informal observation that musicians may generally be more sensitive to audiovisual asynchrony than non-musicians, a topic which has received some attention (e.g., Vatakis and Spence 2006a, b). In a recent study, Lee and Noppeney (2011) compared non-musicians and amateur pianists to test perception of audiovisual simultaneity in music and speech. They found that, compared to non-musicians, those with some piano training had a narrower window of perceived synchrony for piano stimuli, although this did not significantly transfer to perceived audiovisual simultaneity in speech. This lack of transfer effect from piano training to speech processing (Lee and Noppeney, 2011) is somewhat surprising given other research showing extension of musical experience to speech processing (e.g., Wong et al, 2007). A possible explanation stems from an indication by Vatakis and Spence (2006a) that sensitivity to audiovisual synchrony may differ for piano on the one hand, and guitar stimuli on the other. In particular, testing non-musicians they suggest that for piano videos temporal order judgments were more sensitive when the video signal preceded the audio, whereas for guitar stimuli, sensitivity was greatest when the audio preceded the video signal, as has been broadly observed for audiovisual speech processing. These findings suggest that the lack of transfer effect in Lee and Noppeney (2011) may be related to the use of piano stimuli, and that the similarity in audiovisual asynchrony perception for guitar and speech may make guitar stimuli a more sensitive baseline for studying the transfer of musical experience to perception of audiovisual synchrony in speech. In the current study, highly skilled musicians were compared with non-musicians in an audiovisual simultaneity judgment task with audiovisual syllables (/ba/) and audiovisual music stimuli (guitar strum). If sensitivity in the perception of audiovisual synchrony transfers from music experience to speech, musicians relative to non-musicians are expected to be more sensitive to audiovisual asynchrony for both guitar and speech, with a narrower window of synchrony and in particular, an increased sensitivity to an audio signal preceding the video. Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 2

3 METHOD Participants Participants included 19 young adults (20-30 years) of which nine were musicians (4 male, 5 female; M=23 years, SD=1.4 years) and ten were non-musicians (7 male, 3 female; M=24, SD=3.3). All participants had Norwegian as their native language. Participants completed a questionnaire and were pretested to ensure righthandedness (variant of the Edinburgh Handedness Inventory, Oldfield, 1971), normal to corrected visual acuity (Snellen test) and color vision (Ishihara, 1974), as well as normal hearing (PTA threshold of 15dB HL for 250 to 4000 Hz, British Society of Audiology, 2004). Participants all reported less than one year of weekly dance training. The musicians were registered students in Music Performance Studies or Musicology at NTNU where admission requires meeting strict criteria on theoretical and practical musical evaluations in addition to advanced musical skills on a primary and secondary instrument. None of the musicians had voice as their main instrument. Non-musicians had no more than the one year of weekly musical experience common in elementary school. On a ten-point scale, musicians self-reported a very strong interest in music (M=9.9) whereas non-musicians reported having a neutral interest in it (M=5.9). Stimuli Stimuli included audiovisual speech materials and audiovisual music materials. All were recorded in a soundinsulated studio in the Speech Lab, at the Department of Psychology, NTNU. The stimuli are illustrated in Figure 1. The audiovisual speech materials were based on recordings of the syllables /ba/ and /ga/ produced by a young adult female native speaker of Norwegian. The audiovisual music materials were based on recordings of a strum of the top string of a guitar and xylophone stroke. /ga/ and the xylophone stroke are not addressed in the current paper. Both sets of recordings were made using a PDWF800 Sony XDCAM HD422 camcorder and an external Røde NT1-A microphone. The visual dimensions and information ratio of the speech and music materials were comparable. The videos had a resolution of 1133x850 pixels. Audio was adjusted to 68 db and edited in Praat 5.1, then aligned with the videos using Avid Media Composer (v. 6.0). All stimuli were cut to 35 frames where each frame was 40 ms. (1400 ms). All stimuli had the burst (from speech or strum) in the 13 th frame. Both for the speech and music materials, 23 levels of audiovisual alignment were implemented with 40ms (1 frame) steps, from -440 ms audio-lead to 440 ms audio-lead. (a) (b) FIGURE 1. The starting frame from the (a) /ba/ videos and (b) guitar strum videos Procedure The pretests and the main experiment were carried out in the Speech Lab at the Department of Psychology Department, NTNU. In the main experiment up to 5 participants were tested at a time in a partially sound-insulated room, where each participant was seated approximately 50 cm in front of a 24 imac in a four-legged chair to reduce movement. The five computers were set to have the same brightness and sound level (approximately 68 db). A simultaneity judgment task was set up and data collection was carried out using Superlab (v 4.5). In a total of 736 trials, eight randomized repetitions of the 92 stimuli (23 audiovisual alignments x 4 stimulus types) were blocked by stimulus type. Speech and music blocks were alternated, but otherwise were presented in random order. In each trial, a blank screen was presented for 1 s., followed by an audiovisual stimulus. The video was 1133x850 pixels, presented in the center of the monitor. The audio was presented binaurally over AKG K273 studio headphones at 68 ±1 dba. The participants task was to keep focused on the center of the monitor and indicate as Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 3

4 quickly as possible, using a Cedrus RB540 response box, whether the audio and visual components of the stimulus were synchronized (sync, async). Participants had up to 2 s. from the start of the stimulus to respond. The participant s simultaneity judgments were recorded. Between blocks participants had 30-second breaks. The pretests and simultaneity judgment task each took approximately 30 minutes, for a total experiment time of approximately 1 hour. Although two speech stimuli (/ba/ and /ga/) and two music stimuli (guitar strum and xylophone stroke) are included in the experiment, only the /ba/ and guitar strum are addressed here. RESULTS Data from each participant for /ba/ and the guitar strum were plotted with the percentage of responses as a function of audiovisual alignment using Sigma Plot (v.12). As illustrated in Figure 2, a Gaussian curve was fit to the data where the point of subjective synchrony (PSS), audio-lead threshold (ALT), video-lead threshold, and full width of the half maximum (FWHM) were identified (see Conrey and Pisoni, 2006; Hay-McCutcheon et al, 2009). The point of subjective synchrony (PSS) is defined as the x-value for the peak of the Gaussian curve. The audio-lead threshold (ALT) is the x-value which is less than the PSS (i.e. on the left side of the curve) where the y-value is 50%. The video-lead threshold (VLT) is the x-value which is greater than the PSS (i.e on the right side of the curve) where the y-value is 50%. The full width of the half maximum (FWHM) corresponds to the synchrony window, and is the absolute value of the summation of ALT and VLT. (Audio lead) Audiovisual alignment (Video lead) FIGURE 2. Modal Gaussian-fitted curve for responses where the percentage of synchronous responses are plotted as a function of the level of audiovisual alignment. Negative values having audio-lead and positive values have visual lead. The point of subjective synchrony (PSS), the audio-lead threshold (ALT), video-lead threshold (VLT) and full width of the half maximum (FWHM), corresponds to the synchrony window, are indicated in the figure. A mixed analyses of variance was carried out with music experience (musicians, non-musicians) as a between subjects variable, stimulus type (/ba/, guitar strum) as a repeated measure, and PSS, ALT, VLT and FWHM as dependant variables. Results are presented in Figures 3 and 4. As illustrated in Figure 3, simultaneity responses for the speech and music stimuli differed significantly in ALT [F(1,17)=104.66, p=.001], VLT [F(1,17)=151.01, p=.001], and PSS [F(1,17)=126.10, p=.001]. Response for /ba/ were more centered around the point of physical synchrony than responses for the guitar strum, which consistent with Vatakis and Spence (2006b), was shifted in favor of video-lead responses. Notably, no interactions between stimulus type and group were observed, demonstrating that the musicians and non-musicians did not differ in their responses to the two types of stimuli. A comparison of simultaneity judgments from the musicians and nonmusicians, illustrated in Figure 4, shows that responses from both groups had less temporal sensitivity to stimuli with video-lead than audio-lead which is broadly consistent with previous research (e.g., Conrey and Pisoni, 2006; Hay-McCutcheon et al., 2009). No significant between-group difference was found for VLT or FWHM. However, both for the speech and music stimuli, musicians, compared to non-musicians, had an ALT [F(1,17)=11.84, p=.003] which is significantly closer to Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 4

5 the point of physical synchrony as well as a significantly later PSS [F(1,17)=7.21, p=.016], likely related to the shifted ALT. FIGURE 3. Normalized percent simultaneity responses for /ba/ (solid line) and the guitar strum (broken line) averaged across participants and plotted as a function of audiovisual alignment. (Audio lead) Audiovisual alignment (Video lead) (Audio lead) Audiovisual alignment (Video lead) (Audio lead) Audiovisual alignment (Video lead) (a) (b) FIGURE 4. Normalized percent simultaneity responses averaged across non-musicians (solid line) and musicians (broken line) and plotted as a function of audiovisual alignment for (a) the guitar strum and (b) /ba/. DISCUSSION AND CONCLUSIONS Overall results leads to a non-significant tendency for a narrower window of synchrony for musicians than nonmusicians. Although no reliable difference was observed for FWHM, i.e. the width of the synchrony window, as predicted, results for the audio-lead threshold reflect a significantly different perception of audiovisual simultaneity between skilled musicians and non-musicians, with musicians showing a greater sensitivity to audio-lead than nonmusicians. Furthermore, contrary to Lee and Noppeney (2011), the musicians greater sensitivity for audiovisual synchrony spans both guitar and speech stimuli, indicating a transfer from musical experience to speech processing. Findings are consistent with predictions that cross-modal temporal experience increases threshold acuity for audiolead, but not for video-lead, and also support theories suggesting greater efficiency with relevant experience. Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 5

6 ACKNOWLEDGMENTS The authors thank the Music Department, NTNU for collaboration in contacting relevant music students for the study, and all the participants for their willingness to contribute their time and simultaneity responses to this study. This study was carried out as part of a course project in the spring semester of 2012 supervised by the first author with assistance from the second author whose previous research this current study is based on. The names of the third through eighth authors are in alphabetical order and do not reflect the authors contribution. REFERENCES British Society of Audiology. (2004). Recommended procedure: Pure tone air and bone conduction threshold audiometry with and without masking and determination of uncomfortable loudness levels, Last viewed June 3, Boersma, P., and Weenink, D. (2009). PRAAT: doing phonetics by computer (Version 5.1) [Computer program], (retrieved February 8, 2009) Conrey, B., and Pisoni, D. B. (2006). Auditory-visual speech perception and synchrony detection for speech and nonspeech signals, J. Acoust. Soc. Am. 119, Grant, K., Greenberg, S., Poeppel, D., and van Wassenhove, V. (2004). Effects of spectro-temporal asynchrony in auditory and auditory-visual speech processing, Semin. Hear. 3, Grant, K., and Seitz, P. (1998). Measures of auditory-visual integration in nonsense syllables and sentences, J. Acoust. Soc. Am. 104, Hay-McCutcheon, M., Pisoni, D., and Hunt, K. (2009). Audiovisual asynchrony detection and speech perception in hearingimpaired listeners with cochlear implants: A preliminary analysis, Int. J. Audiol. 48, Ishihara, S. (1974). The series of plates designed for a a test for colour-blindness. Tokyo: Kanehara Shuppan. Keetels, M., and Vroomen, J. (2012). Perception of synchrony between the senses, in Frontiers in the Neural Bases of Multisensory Processes, edited by M. M. Murray and M. T. Wallace, (Taylor and Francis Group, London), pp McGrath, M., and Summerfield, Q. (1985). Intermodal timing relations and audio-visual speech recognition by normal-hearing adults, J. Acoust. Soc. Am. 77, Oldfield, R. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9, Petrini, K., Dahl, S., Rocchesso, D., Waadeland, C. H., Avanzini, F., Puce, A., and Pollick, F. (2009). Multisensory integration of drumming actions: musical expertise affects perceived audiovisual asynchrony, Exp. Brain Res. 198, Powers, A. 3rd., Hillock, A., and Wallace, M. (2009). Perceptual training narrows the temporal window of multisensory binding, J. Neurosci. 29, Smeele, P. M. T., Sittig, A., van Heuven, V. (1994): "Temporal organization of bimodal speech information", In ICSLP-1994, Stekelenburg, J. J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19(12), Sumby, W. H., and Pollack, I. (1954). Visual contribution to speech intelligibility in noise, J. Acoust. Soc. Am. 26, Summerfield, Q. (1992). Lipreading and audio-visual speech perception, Philos. Trans. R. Soc. London, Ser. A, van Wassenhove, V., Grant, K. W., and Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. PNAS, 102(4): Vatakis A, and Spence C (2006a). Audiovisual synchrony perception for speech and music using a temporal order judgment task. Neurosci Lett 393:40 44 Vatakis A, and Spence C. (2006b). Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111: Welch R.B., Warren D.H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, Wong PC, Skoe E, Russo NM, Dees T, Kraus N (2007) Musical experience shapes human brainstem encoding of linguistic pitch patterns. Nat Neurosci 10: Proceedings of Meetings on Acoustics, Vol. 19, (2013) Page 6