An a ly s i s -By-Sy n t h e s i s of Ti m b r e, Timing, a n d Dy n a m i c s

Transcription

1 Analysis-By-Syntesis of Timbre, Timing, Dynamics 265 An a ly s i s -By-Sy n t e s i s of Ti m b r e, Timing, a n d Dy n a m i c s in Expressive Clarinet Performance Mat i e u Ba r t e t, Pilippe De pa l l e, Ric a r d Kr o n l a n d -Ma r t i n e t, a n d Sø lv i Ys ta d CNRS Laboratoire de Mécanique et d Acoustique, Marseille, France in a p r e v i o u s s t u d y, m e c a n i c a l a n d e x p r e s s i v e clarinet performances of Bac s Suite No. II and Mozart s Quintet for Clarinet and Strings were analyzed to determine weter some acoustical correlates of timbre (e.g., spectral centroid), timing (intertone onset interval), and dynamics (root mean square envelope) sowed significant differences depending on te expressive intention of te performer. In te present companion study, we investigate te effects of tese acoustical parameters on listeners preferences. An analysis-by-syntesis approac was used to transform previously recorded clarinet performances by reducing te expressive deviations from te spectral centroid, te intertone onset interval and te acoustical energy. Twenty skilled musicians were asked to select wic version tey preferred in a pairedcomparison task. Te results of statistical analyses sowed tat te removal of te spectral centroid variations resulted in te greatest loss of musical preference. Received September 11, 2008, accepted May 4, Key words: timbre, timing, dynamics, expressive clarinet performance, preference Analysis-By-Syntesis of Timbre, Timing, and Dynamics in Expressive Clarinet Performance Te model for musical communication proposed by Kendall and Carterette (1990) in te context of traditional Western art music describes a musical performance as an act during wic a performer transforms te composer s notational signals into acoustical signals tat must be decoded by a listener. Weter tis act of performance can be said to constitute an (aestetic) interpretation depends on ow te performer translates te musical notations into sounds. As a matter of fact, te notion of interpretation is defined as te act of performance wit te implication tat in tis act, te performer s judgment and personality necessarily ave teir sare (Scoles, 1960). Previous studies on musical performance ave sown tat performers use timing and intensity variations to play expressively (see Gabrielsson, 1999, for a review). However, less attention as been paid so far to timbre (see e.g., Juslin & Laukka, 2003, for a review), te perceptual attribute of sound tat was described by Seasore (1938/1967) as te most important aspect of tone, and wic introduces te largest number of problems and variables. Tis article is te second part of a study designed to test weter timbre variations are linked to te expressive intentions of performers and to te musical preferences of listeners. Previous analyses of recorded clarinet performances elped to determine te acoustical correlates of a performer s expressive variations in timbre, timing and dynamics (Bartet, Depalle, Kronland-Martinet, & Ystad, 2010). Te perceptual effects of tese acoustical parameters were investigated in te present study in terms of musical preferences, using an analysis-bysyntesis approac (Risset & Wessel, 1999). Modeling Musical Interpretation In addition to measuring musical performances, many autors ave focused on modeling musical expressiveness. Detailed istorical reviews of models for musical interpretation ave been publised by Widmer and Goebl (2004) and DePoli (2006). Te strategies most commonly used for tis purpose are analysis-by-measurement (see for example, Todd, 1992, 1995; Windsor, Desain, Penel, & Borkent, 2006), analysis-by-syntesis, and music teory knowledge (see for example, Friberg, 1995), macine learning (see for example, Goebl, Pampalk, & Widmer, 2004; Tobudic & Widmer, 2003), and combinations of tese metods. Some of tese models are predictive and account for te act of performance at its source, based on te assumption tat interpretation can be described as a set of generative rules, wile oters are based on descriptive parameters and account for te effects of interpretation, i.e., te expressive deviations. All of te above Music Perception v o l u m e 28, issue 3, p p , issn , e l e c t ro n i c issn b y t e re g e n t s of t e university of california. a l l r i g t s re s e rv e d. p l e a s e di r e c t al l re q u e s t s fo r permission t o p o t o c o p y or re p r o d u c e ar t i c l e co n t e n t t r o u g t e university of california pr e s s s r i g t s an d permissions we b s i t e, t t p ://w w w.uc p r e s s j o u r n a l s.co m /re p r i n t i n f o.as p. DOI: /m p

2 266 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad models focus mainly on timing, dynamics, articulation, and intonation, wereas timbre as often been neglected. It seems necessary to model canges of timbre in order to efficiently simulate expressive performances, especially in te case of self-sustained instruments suc as te clarinet or te violin, wit wic te sound continues to be controlled after te onset of a note (Kergomard, 1991). In teir model, Canazza, Rodá, and Orio (1999) and Canazza, De Poli, Drioli, Rodá, and Vidolin (2004) took timbre variations into account in addition to te rytmic and dynamic aspects of musical performance. Timbre, A Multidimensional Perceptual Attribute of Complex Tones Timbre is by definition te perceptual attribute tat allows one to distinguis tones of equal pitc, loudness, and duration (ANSI, 1960). According to Handel (1995), te identification of timbre depends on on our ability to recognize various pysical factors tat determine te acoustic signal produced by musical instruments (called source mode of timbre perception in Hajda, Kendall, Carterette, & Harsberger, 1997), as well as to analyze te acoustic properties of sound objects perceived by te ear, wic as traditionally been modelled as a time-evolving frequency analyser (called interpretative mode of timbre perception in Hajda et al., 1997). Timbre terefore involves two complementary facets, as it relates to bot te identity and te quality of sound sources. No general models for timbre ave been developed so far. However, te seminal researc by Grey (1977), Wessel (1979), Krumansl (1989), Kendall and Carterette (1991) and McAdams, Winsberg, Donnadieu, De Soete, and Krimpoff (1995) developed geometrical models for timbre tat represent te organization of perceptual distances (te so-called timbre space), measured on te basis of dissimilarity judgments between tones wit equal pitc, loudness, and perceived durations. Two- to fourdimensional timbre spaces ave often been found in dissimilarity studies between various natural or syntetic tones corresponding to orcestral instruments (Caclin, McAdams, Smit, & Winsberg, 2005). Te main acoustic correlates of timbre-space dimensions are te attack time (correlated wit te rate of energy increase in a sound, see Krimpoff, McAdams, & Winsberg, 1994), and te spectral centroid (te mean of te spectral energy distribution, see Grey & Gordon, 1978). Te spectral flux (measure of te fluctuation of te spectrum over time, see McAdams et al., 1995), and te spectral irregularity (an index to te disparities between te armonic components, see Krimpoff et al., 1994) are examples of oter timbre descriptors tat ave often been proposed as correlates of timbre space dimensions. On te Role of Timbre in Musical Performance In Western traditional music wic is te type of music wit wic most studies on musical performance ave dealt te role of timbre seems to be understimated compared to tose of rytm, pitc, and dynamics. Tis is notably revealed by te Western traditional notation system tat almost omits timbre except in te references to te instrument (typological aspect of timbre), or in certain musical terms (e.g., dolce, duramente, con brio, legato), wic can refer directly or indirectly to morpological aspects of timbre (Risset, 1994). It is wort noting tat some igly complex timbre notation systems including more tan one undred symbols ave been developed for some traditional Cinese and Japanese instruments suc as te Cin, an ancient Cinese seven-string lute (see Traube, 2004), and te Sakuaci, a Japanese bamboo flute. In te music played wit tese instruments, timbre terefore seems to be given as muc importance as rytm or pitc. It was not until te contemporary period and te new possibilities provided by digital sounds tat timbre was placed at te foreground of Occidental music by composers suc as Varèse, wo proposed not only to compose wit sounds but to compose te sounds temselves (Risset, 1994). Te lack of notational information relating to timbre in traditional Western tonal music certainly does not mean tat performers do not use timbre as a parameter to express feelings and emotions, owever. In a previous experiment (Bartet et al., 2010), we investigated te acoustical parameters accounting for expressiveness in clarinet playing. To address tis issue, mecanical and expressive performances of excerpts from te Classical and Baroque repertoire were recorded and analyzed. Several temporal and spectral parameters were computed to caracterize te acoustical features of te performer s interpretations. Statistical analysis of te data sowed significant effects of te expressive intention on tree timbre descriptors adapted to te clarinet (te attack time [AT], te spectral centroid [SC], and te odd/ even ratio [OER], see Bartet, 2008), te intertone onset interval (IOI), quantifying te durations of te tones (see Repp, 1992), and te root mean square envelope (ENV), caracterizing te variations of te acoustical energy. Tese results sowed tat canges in te timbre descriptors (AT, SC, OER), timing descriptor (IOI), and dynamics descriptor (ENV) occurred wen te performer played in a more expressive way. Among te tree timbre descriptors of clarinet tones (attack time, spectral centroid, odd/ even ratio) tat were analyzed in (Bartet et al., 2010), te spectral centroid was found to be te main predictor of te performer s expressive intentions (among all descriptors, SC was te one tat most frequently differentiated between mecanical and expressive performances).

3 Analysis-By-Syntesis of Timbre, Timing, Dynamics 267 Figure 1. Exploration of te acoustical correlates of musical expressiveness based on te analysis-by-syntesis approac. We terefore decided to focus on tis timbre descriptor in te present experiments. Te variations of acoustical features caracterizing recorded clarinet performances were modified using signal processing tecniques in order to furter assess te effects of tese canges on te musical preferences of listeners. Metod General Metodology We developed a general metodology to explore te role of timbre in musical performance, based on te analysisby-syntesis paradigm (Risset & Wessel, 1999). Te metod, wic is summarized in Figure 1, comprises four steps: te extraction of a representation from te signal (analysis), te transformation of tis representation in te frequency domain, te conversion of te modified representation back to te time domain (syntesis), and te analysis of te perceptual effects induced by te transformation. In order to identify te acoustical parameters tat contribute most to musical aestetic judgments, we investigated te effects of reducing te performer s original expressive deviations on preferences expressed by listeners in a paired-comparison task. Stimuli sound corpus Expressive clarinet performances of te Allemande movement of Bac s Suite No. II (BWV 1008) and te Largetto movement from Mozart s Quintet for Clarinet and Strings (KV 581) were recorded in an anecoic MP2803_03.indd 267 camber (see Bartet et al., 2010, for furter details). Te scores of te excerpts and te sound examples related to te study are available at: ttp:// cnrs-mrs.fr/~kronland/interpretation_perceptual. Te first musical prases in tese pieces were selected to generate te stimuli. Tese excerpts were cosen so tat tey would be sufficiently long to ave a musical meaning (a musical prase), but sufficiently sort for a pairedcomparison task (10 to 15 s excerpts; see (Gabrielsson & Lindstrom, 1985) for a discussion on te stimuli durations in paired comparisons). Analysis-Syntesis Model Te musical sequences were resyntesized using an additive-based syntesis model. Te sound model decomposes an audio signal in to a deterministic part, consisting of a sum of quasi-sinusoidal components plus a residual part. A tone s(t) can tus be written as follows: H s(t ) = A (t )cos[φ (t )]+ r(t ) =1 (1) t φ (t ) = 2π f (τ )dτ + φ (0) 0 were A(t), f(t), and f(t) are te instantaneous amplitude, pase, and frequency, respectively, of te t among H sinusoids, f(0) is te initial pase, and r(t) is te residual. Tis metod is particularly suitable for canging te caracteristics of te tones related to timbre and timing, for example, since sounds are reconstructed as a superposition of partial components, te frequency and amplitude of wic can be individually controlled. 1/12/11 6:03:56 PM

4 268 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad In addition to teir armonic structure, clarinet tones contain ancillary noises (for instance breat and key noises) tat also contribute to te identity of te instrument. Te latter noises are partly contained in te residual. Te residual was obtained by performing a time-domain subtraction between te original sequence and te resyntesized deterministic part wit no transformations. Resyntesized sequences were ten obtained by juxtaposing te resyntesized tones. Tis procedure gives a ig-quality resyntesis of te clarinet performances (cf. sound examples 1 to 3). Transformations Recordings of a players original performances can be transformed by appropriately manipulating te control parameters in te model. In order to assess te perceptual effects of spectral centroid (SC) variations, intertone onset interval (IOI) deviations, and acoustical energy (ENV) variations, tree transformations were carried out to independently modify tese parameters. Spectral centroid freezing (T T ). A transformation acting on te spectral centroid was designed to control te sape of te tones spectral centroids witout affecting teir acoustical energy. Te metod used for tis purpose was based on te transformation described by McAdams, Beaucamp, and Meneguzzi (1999), wic consists in eliminating te spectral flux (measure of te fluctuation of te spectrum over time), wile leaving te RMS envelope intact. As a matter of fact, te removal of te spectral flux corresponds to canceling te time-dependent variations in te spectral envelope s sape, tus cancelling te spectral centroid variations. As te drastic elimination of all spectral centroid variations may cause tones to sound too artificial, te microfluctuations in te instantaneous amplitudes were preserved by separating te slow variations in te instantaneous amplitudes L (t), defined as te amplitude variations of frequencies below 10 Hz, from te fast variations H (t), defined as te amplitude variations of frequencies above 10 Hz. Te modified instantaneous amplitude Ã(t) of te t component of a given tone is obtained as follows: A () t = β () t ENV() t β () t = A + H () t H = 1 [ A H()] t 2 + (2) were ENV(t) is te RMS envelope of te tone, and b (t) is a term computed to fluctuate around te time-averaged amplitude A of te t armonic. A was determined during te sustained part of te tone. Since te ig-frequency fluctuations H (t) are small in comparison wit A, te term b (t) is almost constant wit time. Te sape of te tones new instantaneous amplitudes à (t) is tere fore very similar to tat of te RMS envelope of te tone ENV(t) wit various scale factors. Figure 2 sows te instantaneous amplitudes of a clarinet tone before and after te transformation. Te modified spectral centroid is almost frozen over time, altoug it varies quickly around te mean value of te initial spectral centroid calculated during te sustained part of te tone. Te application of tis transformation to one of te clarinet performances is given, as example, in Figure 3. Te original spectral centroid variations are presented in Figure 3(a), and te modified ones in Figure 3(b). Intertone onset interval deviation cancellation (T R ). We designed a transformation for replacing te effective intertone onset intervals (as played by te performer) wit te nominal IOIs given by te transcription of tescore notations. Tis was done by applying time-scale modifications to te instantaneous frequencies and amplitudes of te tone s components. Tese canges were applied only to te sustained and release portions of te tones, sparing te original attack, wic is known to be an important attribute of timbre. Te time dilation/ contraction coefficient a was computed for eac tone as follows: IOI α = IOI eff nom AT AT (3) were IOI eff and IOI nom denote te tone s effective and nominal IOIs, and AT is te attack time. For eac tone, after te attack, te instantaneous frequencies f (t) and amplitudes à (t) were transformed as follows: f () t = f ( αt) A () t = A ( αt) (4) Figures 3(c) and 3(d) sow te effect of te transformation on te IOI deviation descriptor ΔIOI. Tis transformation yields mecanical performances tat are exactly in line wit te timing indications given on te score. Compression of te dynamics (T D ). In order to reduce te variations in te acoustical energy, we used a dynamic range controller serving as a compressor and limiter (see Zölzer, 1997). Te signal s input level was determined via an envelope follower based on peak measurements. A gain factor was ten used to adjust te amplitude of te input signals. Tis compressor/limiter was controlled

5 Analysis-By-Syntesis of Timbre, Timing, Dynamics 269 (a) Original Instantaneous Amplitudes (b). Transformed Instantaneous Amplitudes Figure 2. Transformations of te instantaneous amplitudes (armonics 1 to 7): (a) original clarinet tone - (b) after te spectral centroid freezing. by te parameters generally used wit devices of tis kind, i.e., by te tresolds and slope of te limiter, te tresolds and slope of te compressor, and te attack and release times. Altoug tis metod is based on nonlinear processing procedures liable to cause armonic distortions, te control parameters of te dynamic range controller were carefully selected to prevent te occurrence of any audible canges in te timbre. Te attack and release times were bot set at 10 ms. As compression and limiting procedures lead to

6 270 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad (a) Original Spectral Centroid (b) Transformed Spectral Centroid (c) Original IOI Deviations (d) Transformed IOI Deviations (e) Original RMS Envelope (f) Transformed RMS Envelope Figure 3. Original (left) and transformed (rigt) expressive patterns for a clarinet performance of te Mozart excerpt.

7 Analysis-By-Syntesis of Timbre, Timing, Dynamics 271 canges in te signal levels, te loudness of eac modified sequence was equalized to a constant value. Examples of original and transformed RMS envelopes are given in Figures 3(e) and 3(f), respectively. As was to be expected, te range of variation of te acoustical energy is muc smaller in te modified version. To avoid effects of pitc on te preferences of te listeners, te fundamental frequencies of te tones were set at teir mean values computed in te sustained parts, for eac transformation. Te instantaneous frequencies of te tones components were terefore set at f 0, were f 0 denotes te mean fundamental frequency of te tone. We cecked tat te frequency canges were only weakly perceptible in te musical excerpts selected (cf. sound example 4). Design of te Stimuli Te tree basic transformations (T T, T R, T D ) and teir four combinations (T TR, T TD, T RD, T TRD ) were applied to te expressive clarinet performances. As te original performances were recorded in an anecoic camber, a sligt reverberation was added to te resyntesized versions to make tem sound more natural. Te eigt stimuli listed in Table 1 terefore were generated for eac musical excerpt. Te stimuli associated to te Bac and Mozart excerpts correspond to te sound examples 5 to 12, and 13 to 20, respectively. Participants Given te relatively demanding requirements of te auditory discrimination task, in wic te participants ad to rate te interpretations wit various levels of expression, te experiment was carried out wit skilled musicians (14 males, 6 females; age = years) as listeners. Most of te participants were students in musicology practicing various musical instruments suc as clarinet, guitar, piano, violin, etc., wo were participating Table 1. Description of te Stimuli. Stimuli Transformation description M 0 No transformation M T Freezing of te spectral centroid (T T ) M R Canceling of te IOI deviations (T R ) M D Compression of te dynamics (T D ) M TR Combination of T T and T R M TD Combination of T T and T D M RD Combination of T R and T D M TRD Combination of T T, T R and T D in improvisation worksops at te GRIM (Groupe de Recerce et d Improvisations Musicales, Marseille). Apparatus Te experiment was carried out in an audiometric cabin. Te user interface was implemented in te Matlab environment. 1 Te sound files were stored on te ard drive of an Apple imac G4 computer and delivered to te participants via a STAX SRM-310 eadpone system. Procedure Participants were asked to select wic stimuli tey preferred in a paired comparisons task. Tey first underwent a training pase in order to become familiar wit te task. In order to assess te influence of te musical excerpt being used, eac participant attended two sessions, one wit te Bac sequences as stimuli, and te oter wit te Mozart sequences as stimuli. Te order of te two sessions (denoted Bac and Mozart ) was counterbalanced across participants. At te end of te test, te participants were asked to complete a questionnaire specifying wat criteria tey ad used to assess te recordings. During te experiment, participants listened to several successive pairs of clarinet performances separated by a 1 s interval. Tey could listen to eac performance as many times as tey wised. At eac trial, participants were asked to indicate wic version tey preferred. All te possible combinations (28) of te eigt stimuli (te tree basic transformations, teir four combinations and te original performance) were presented to eac participant. Te witin-pair order and te order of te pairs were randomized. Eac session lasted approximately 20 minutes. Results and Discussion Presentation of te Perceptual Data Te responses of eac participant m are presented in te form of a preference matrix P m. Te elements of te preference matrix, denoted P m (i,j), designate weter stimulus i was preferred to stimulus j. Te global preference matrix P of te sample is defined as te sum of te individual preference matrices P m. Wit eac participant, te various performances were given preference scores S m (i), depending on te number of times tey were preferred to te oters; wit eac performance i, tis corresponds to summing te preference matrix elements P m (i,j) across te columns. Te scores range from 0 to 7 (times preferred). Te mean 1 ttp://

8 272 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad preference scores, denoted S(i), were computed on te basis of te preference scores S m (i), associated wit te ratings of te participants. An alpa level of.05 was used for all statistical tests. Sample Homogeneity Te degree of agreement among te participants was computed using te Kendall coefficient of agreement u for paired comparisons, as described by Siegel and Jon Castellan Jr. (1988). Tis nonparametric measure of association can be written as follows: N N Pij (,) s s 2 i j C u = = 1 = Nm Ns C C 2 2 (5) were C k n denotes te binomial coefficient, N s is te total number of stimuli, and N m is te total number of participants. Here, N s = 8 and N m = 20. Wen N m is even, u 1 can range from to 1, te latter meaning tat tere is N m 1 complete agreement among te participants. Siegel and Jon Castellan Jr. (1988) defined an index W T based on u, wic can range from 0 to 1. Te statistic u can be taken to be an estimate for a population parameter ν, wic stands for te true degree of agreement in te population. We tested te null ypotesis (H 0 : n = 0) tat tere was no agreement among te participants against te alternative (H 1 : n 0) tat te degree of agreement was greater tan wat one would ave expected ad te paired comparisons been done at random. As te total number of participants was large (N m > 6), we used a large-sample approximation of te sampling distribution, asymptotically distributed as a χ 2 distribution (df = 28). Te values of u and W T calculated wit bot te Bac and Mozart excerpts can be found in Table 2. Tese results sow tat te agreement among te participants preferences was significantly iger tan cance bot wit te Bac (u = 0. 58, p <.001) and Mozart (u = 0.52, p <.001) sequences. Analyses of Variance (ANOVA) In order to assess te influences of te musical excerpts (two modalities) and te transformations (eigt modalities) on te participants preferences, te preference Table 2. Coefficients of Agreement Among te Participants at te Bac and Mozart Sessions. Session df u W t X 2 Bac (p <.001) Mozart (p <.001) scores S m (i) were subjected to a two-way repeated measures analysis of variance (ANOVA). Te results of te ANOVA, wic are presented in Table 3, sow tat te differences of preference scores between te two excerpts were not sufficiently large to exclude te possibility tat tey migt be due to cance. Indeed, te effect of te musical excerpt on te preference scores was not found to be significant (te mean values of te preference scores were identical for bot excerpts). Te interaction between te musical excerpt and te transformations was also not significant, F(7, 133) =1.05, p =.40. Conversely, te effect of te transformations on te preference scores was igly significant, F(7, 133) = , p <.001. In order to compare te between-transformation effects, multiple comparison tests (Tukey Honestly Significant Difference tests) were conducted for eac excerpt. Tis procedure determined te significant differences existing between te mean preference scores in eac 2 by 2 combination between te various transformations. Te results of te multiple comparison procedure are presented in Tables 4 and 5. Te preference scores associated wit te various performances are described by te box-and-wisker diagrams sown in Figure 4. For bot te Bac and Mozart excerpts, te version M 0 was te most frequently preferred rendering (see Figure 4). Tis is not surprising, since te expressive deviations associated wit timbre, timing, and dynamics ad not been removed from tis version. It is also not surprising tat te performance M TRD to wic te tree basic transformations were applied was te least preferred on average. Te removal of te IOI deviations (T R ) was te transformation tat resulted on average in te least loss of musical preference in te case of bot excerpts (see Figure 4). Te fact tat te IOI deviations removal ad minor effects on te musical preferences does not mean tat te IOI deviations are not an important feature of preference, but means tat teir effect was weak compared to tat of te acoustical energy and te spectral centroid variations for tese excerpts. Table 3. Results of te Two-Way Repeated Measures Analysis of Variance of te Preference Scores. Source df F p Excerpt Transformation *** <.001 Excerpt Transformation Error 133 (1.11) Note. Te result enclosed in parenteses represent te mean square error. *p <.05, **p <.01, ***p <.001

9 Analysis-By-Syntesis of Timbre, Timing, Dynamics 273 Table 4. Results of te Multiple Comparison Procedure Witin te Bac Session. M 0 M T M R M D M TR M TD M RD M TRD M *** 3.60 (p =.18) 6.61*** 15.82*** 22.62*** 8.81*** 22.22*** M T 9.61*** 6.61*** 2.60 (p =.59) 9.41*** 4.41* 9.01*** M R 3.00 (p =.40) 12.21*** 19.02*** 5.21** 18.62*** M D 9.21*** 16.02*** 2.20 (p =.78) 15.62*** M TR 6.81*** 7.01*** 6.41*** M TD 13.82*** 0.40 (p = 1) 13.41*** M RD Note. Te table presents te results of te pairwise multiple comparison procedure associated to te eigt transformations (Tukey HSD tests). Te values of te studentized range statistic q are reported; *p <.05, **p <.01, ***p <.001. However, it is wort noting tat te differences of preference between te sequence M R, witout te IOI deviations, and te expressive sequence M 0 were not significant for bot excerpts (see Tables 4 and 5), altoug significant effects of te performer s expressive intentions on te IOI deviations descriptor were found for bot excerpts in our companion study (Bartet et al., 2010). Hence, even if te IOI deviations sowed significant differences from te acoustical point of view, te differences migt ave been too subtle to significantly alter te preferences of te listeners (cf. sound examples 7 and 15). Tis finding proves te interest of an analysis-by-syntesis approac to investigate te perceptual effects of specific acoustical features. In te case of te Bac excerpt, te fact tat te IOI deviations ad minor effects on preference is probably due to te style of te musical piece, wic is an instrumental dance. Listeners migt terefore expect te excerpt to be played wit smaller IOI deviations from nominal durations (i.e., in a mecanical way). In te case of te Mozart excerpt, te fact tat te removal of te IOI deviations ad little effect on te listeners preferences is more surprising, as timing deviations are more likely to occur in slow tempo movements (Largetto in tis case). However, te variance of te preference scores attributed to M R was greater wit te Mozart excerpt tan wit te Bac excerpt. After te removal of te IOI deviations, te compression of te dynamics was te transformation tat ad te least effect on te musical preference. However, te differences between M D and te reference M 0 were significant wit bot excerpts (see Tables 4 and 5). At bot sessions, te performances tat were processed by freezing te spectral centroid (M T, M TD, M TR, M TRD ) were consistently te least preferred (see Figure 4). As sown in Tables 4 and 5, tese versions sowed te most significant differences wit te reference version M 0. Te spectral centroid freezing procedure terefore resulted in a greater loss of musical preference tan te removal of te IOI deviations, or te compression of te dynamics. It is wort noting tat removal of te spectral centroid variations ad more degrading effects tan te removal of te IOI deviation and te dynamic transformations combined (cf. score of M T versus te one of M RD ). Hierarcical Cluster Analysis (HCA) In order to determine wic performances sowed systematic similarities or differences in terms of musical Table 5. Results of te Multiple Comparison Procedure Witin te Mozart Session. M 0 M T M R M D M TR M TD M RD M TRD M *** 1.40 (p =.98) 6.81*** 13.61*** 21.02*** 7.21*** 20.02*** M T 13.21*** 7.81*** 1.00 (p = 1) 6.41*** 5.37** 5.41** M R 5.41** 12.21*** 19.62*** 6.01*** 18.62*** M D 6.81*** 14.22*** 0.60 (p = 1) 13.21*** M TR 7.41*** 6.21*** 6.41*** M TD 13.61*** 1.00 (p = 1) 12.61*** M RD Note. Legend: see Table 4.

10 (a) Bac Session (b) Mozart Session Figure 4. Box and wisker plots of te listeners preference scores at te Bac session (a) and te Mozart session (b). Te various sequences are arranged from left to rigt in decreasing order, based on te median values of te scores. Te lines in te box are at te lower quartile, median, and upper quartile values. Te wiskers sow te range of te rest of te data. Teir lengts was 1.5 times te interquartile range. Outliers are represented by crosses. Te notces on te box give a robust estimate of te uncertainty of te medians. Note tat te number of times eac sequence was preferred ranged necessarily between 0 and 7. preference, te mean preference scores associated to te performances were analyzed using a ierarcical cluster analysis (HCA). Between-performance distances were obtained by computing te Euclidean distances between te mean preference scores S(i). Two different ierarcical clustering metods of te performances were tested: (1) te complete linkage, wic is based on te furtest distance between te elements of eac cluster, and (2) te Ward linkage, wic is based on te increase in variance for te clusters being merged (see e.g., Dillon & Goldstein, 1984). Bot metods returned similar ieracical cluster trees (dendrograms). As sown in Figure 5, wic presents te dendrograms obtained wit te complete linkage metod, te two main clusters associated to te Bac and Mozart sessions were identical. One of tem contains all te performances tat underwent te spectral centroid freezing transformation (M T, M TD, M TR, M TRD ), and te oter contains te remaining performances (M 0, M R, M D, M RD ). Tese results sow tat te spectral centroid freezing transformation induced a drastic cange of musical preference relative to te intertone onset interval deviation cancellation and/or te compression of te dynamics. Tey also sow tat te preference scores of te performances M R, M D, and M RD were systematically closer to te score of te reference M0, in comparison to te scores of te performances tat underwent te timbre transformation. General Discussion Te results sowed tat te preferences of te participants depended on wic acoustical parameters ad been modified (spectral centroid and/or intertone onset interval and/or RMS envelope). Te preference scores of sequences submitted to multiple transformations were lower or, at best, equal to te score of te basic constituent transformation tat was te least preferred. For instance, te score obtained by te sequence M RD, for wic bot te IOI deviations and te dynamics were modified, was lower to tat obtained by te sequence M D, wic was less preferred tan te sequence M R. No significant effect of te musical excerpt on te preferences was observed. Among te seven transformations, te greatest effects observed were tose caused by te freezing of te spectral centroid. Tis transformation results in a greater loss of preference tan te one caused by removal of te IOI deviations or dynamic compression or tese two transformations combined. McAdams et al. (1999) investigated subjects ability to discriminate between various isolated instrumental tones in wic spectrotemporal simplifications ad been made. Among tese simplifications, te freezing of te spectral centroid (induced by te spectral flux freezing process) was found to be most easily discriminated by te listeners. Tis effect was less marked,

11 Analysis-By-Syntesis of Timbre, Timing, Dynamics 275 (a) Bac Session (b) Mozart Session Figure 5. Dendrogram representations of te Hierarcical Cluster Analysis of te between-performance distances (complete linkage metod), for te excerpts from Bac (a) and Mozart (b). owever, wit clarinet and oboe tones since teir spectra generally undergo muc less spectral flux tan most oter instruments, namely te brass instruments (Grey, 1977). Te strong influence of te removal of spectral centroid variations may be due to te fact tat tis descriptor is correlated to one of te main timbre dimensions (Grey, 1977; Krumansl, 1989; McAdams et al., 1995). Altoug analysis/syntesis tecniques allow te control of te spectral centroid independently from oter sound dimensions (suc as te acoustical energy, in particular), tis process is not necessarily acceptable regarding te timbral identity of instrumental tones. However, te participants never preferred te versions in wic bot te spectral centroid and te acoustical energy variations ad been transformed. Te spectral centroid freezing probably alters te original timbre of te instrument so tat it becomes unnatural, or at least different. In tis case, te transformation would do more tan simply remove te expressive deviations of timbre, since it would cange te nature of te instrument. It sould be noted tat te participants did not report tat tey ad relied on timbre identity canges. Most of tem sowed a preference for performances wit lively rater tan static tones, wic refers directly to te presence or absence of variations in te intensity and/or timbre during te tones. Te clarinetist wo ad played te performances perceived tat te transformed sequences were different from is original performances but still tougt tey were played wit a clarinet. Among te possible causal explanations for te differences, e suggested, for instance, tat te instrument migt ave been poorly controlled by te player (e.g., students wo move teir moutpiece), or tat it migt ave been in poor condition (e.g., an old reed), resulting in a general lack of omogeneity in te interpretation. Abeles (1973) developed a clarinet performance adjudication scale and observed tat te timbre of te tones was one of te most important factors used by music teacers to rate clarinet performances. It is terefore not so surprising tat te present participants based teir assessments mainly on te timbre of te clarinet tones. In tese experiments, te intricate process of interpretation was reduced to a simple linear, additive model (SC variations +/ IOI deviations +/ energy variations). Tis is a first step towards reacing a better understanding of te influence of te temporal and spectral parameters related to timbre, timing, and intensity. However, tese parameters may interact at te level of bot performers (in te sound production process) and listeners (in te decoding of te musical signals). For instance, wen listening to is own performance of te excerpt from Mozart s Largetto, te clarinetist was convinced tat e ad deliberately lengtened one of te tones in te sequence, causing te following one to be late. However, te analysis sowed tat te onset of te second tone was perfectly on te beat. Te decrescendo in te first of tese two tones, wic was associated wit a concomitant decrease in brigtness, may ave induced tis feeling of ritardando.

12 276 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad Summary and Conclusions Tis study focused on te perceptual effects of variations in acoustical correlates of timbre, timing, and dynamics on musical preference. To address tis issue, an experimental metod based on te analysis-by-syntesis approac was developed, wic consisted of transforming te expressive content of recorded clarinet performances and assessing te effects of tese canges on listeners musical preferences. Seven transformations were designed to remove or compress te variations in te spectral centroid (a timbre parameter), intertone onset interval (a timing parameter), and acoustical energy, eiter separately or in various combinations. Te statistical analyses carried out on te listeners aestetic judgments sowed tat te spectral centroid freezing transformation most significantly decreased te musical preference of te performances. Tis finding seems to be due to te fact tat tis transformation altered te original timbre of te clarinet tones (te identity of te instrument), as well as drastically affecting te time-evolving spectral sapes, causing te tones to be static and unlively (te quality of te sound). Tese results confirmed tat te performer s coices of timbre, timing, and dynamics variables (see te companion article by Bartet et al., 2010) affect te listener s perception of te musical preference. Te variations in te spectral centroid during tone production seem to be an important feature of preference in te musical message transmitted from performers to listeners. Indeed, in anoter study, we establised tat controlling te time-evolving spectral centroid of tones improved te preference of sampler-based generated sequences (Bartet, Kronland-Martinet, & Ystad, 2008). Tese findings suggest tat it migt be wort developing a general set of rules related to timbre, as previous autors ave done in te case of temporal and intensity deviations (see e.g., De Poli, 2006; Matews, Friberg, Bennett, Sapp, & Sundberg, 2003; Widmer & Goebl, 2004). By taking te variations in te acoustical parameters associated wit timbre into account in computational models of music performance, it migt ten be possible to improve te automatic rendering of musical pieces by computers. Autor Note We would like to tank te clarinetist Claude Crousier for participating in tis project and many fruitful discussions. We are grateful to Mitsuko Aramaki and Henri Burnet from te Institut des Neurosciences Cognitives de la Méditerranée and Jean Pierre Durbec from te Centre d Océanologie de Marseille for teir precious elp. We would also like to tank te reviewers for teir elpful comments and advices. Tis project was partly supported by te Frenc National Researc Agency (ANRJC , sensons ttp:// cnrs-mrs.fr/). Autor Pilippe Depalle is now affiliated wit te Sound Processing and Control Laboratory, Te Sculic Scool of Music, McGill University, Montreal, Canada. Correspondence concerning tis article sould be addressed to Matieu Bartet, CNRS Laboratoire de Mécanique et d Acoustique, 31 cemin Josep-Aiguier, Marseille Cedex 20, France. e-m a i l: bartet@lma.cnrs-mrs.fr References Abe l e s, H. F. (1973). Development and validation of a clarinet performance adjudication scale. Journal of Researc in Music Education, 21, ANSI. (1960). USA Standard Acoustical Terminology. New York: American National Standards Institute. Bar t e t, M. (2008). De l interprète à l auditeur: Une analyse acoustique et perceptive du timbre musical [From performer to listener: An acoustical and perceptual analysis of musical timbre]. Unpublised doctoral dissertation, Université Aix-Marseille II, Marseille, France. Bar t e t, M., De pa l l e, P., Kr o n l a n d -Ma r t i n e t, R., & Ysta d, S. (2010). Acoustical correlates of timbre and expressiveness in clarinet performance. Music Perception, 28, Bar t e t, M., Kr o n l a n d -Ma r t i n e t, R., & Ys ta d, S. (2008). Improving musical expressiveness by time-varying brigtness saping. In R. Kronland-Martinet, S. Ystad, & K. Jensen (Eds.), Sense of sounds (pp ). Berlin/Heidelberg: Springer-Verlag. Ca c l i n, A., McAd a m s, S., Sm i t, B. K., & Wi n s b e r g, S. (2005). Acoustic correlates of timbre space dimensions: A confirmatory study using syntetic tones. Journal of te Acoustical Society of America, 118, Cana z z a, S., De Po l i, G., Dr i o l i, C., Ro d á, A., & Vidolin, A. (2004). Modeling and control of expressiveness in music performance. Proceedings of te IEEE, 92, Cana z z a, S., Ro d á, A., & Or i o, N. (1999). A parametric model of expressiveness in musical performance based on perceptual

13 Analysis-By-Syntesis of Timbre, Timing, Dynamics 277 and acoustical analyses. In Proceedings of te International Computer Music Conference (pp ). Beijing, Cina: International Computer Music Association (ICMA). De Po l i, G. (2006). Algoritms for sound and music computing. Creative Commons Attribution-NonCommercial-SareAlike licence. Retrieved from ttp:// Dispense06/7_espressiveness.pdf Dil l o n, W. R., & Go l d s t e i n, M. (1984). Multivariate analysis. New York: Jon Wiley & Sons. Fri b e r g, A. (1995). A quantative rule system for musical performance. Unpublised doctoral dissertation, Royal Institute of Tecnology, Stockolm, Sweden. Gabrielsson, A. (1999). Te performance of music. In D. Deutsc (Ed.), Te psycology of music (2nd ed., pp ). San Diego, CA: Academic Press. Gabrielsson, A., & Li n d s t r o m, B. (1985). Perceived sound quality of ig-fidelity loudspeakers. Journal of te Audio Engineering Society, 33, Goe b l, W., Pa m pa l k, E., & Wi d m e r, G. (2004). Exploring expressive performance trajectories: Six famous pianists play six Copin pieces. In S. D. Lipscomb, R. Asley, R. O. Gjerdingen, & P. Webster (Eds.), Proceedings of te 8t International Conference on Music Perception and Cognition (pp ). Evanston, IL: Causal Productions. Gre y, J. M. (1977). Multidimensional perceptual scaling of musical timbres. Journal of te Acoustical Society of America, 61, Gre y, J. M., & Go r d o n, J. W. (1978). Perception of spectral modifications on orcestral instrument tones. Computer Music Journal, 11, Haj d a, J. M., Ke n d a l l, R. A., Ca rt e r e t t e, E. C., & Ha r s - b e rg e r, M. L. (1997). Metodological issues in timbre researc. In I. Deliège & J. A. Sloboda (Eds.), Perception and Cognition of Music (2nd ed., pp ). New York: Psycology Press. Han d e l, S. (1995). Timbre perception and auditory object identification. In B. C. J. Moore (Ed.), Handbook of perception and cognition (2nd ed., pp ). San Diego, CA: Academic Press. Jus l i n, P. N., & Lau k k a, P. (2003). Communication of emotions in vocal expression and music performance: Different cannels, same code? Psycological Bulletin, 129, Ken d a l l, R. A., & Ca rt e r e t t e, E. C. (1990). Te communication of musical expression. Music Perception, 8, Ken d a l l, R. A., & Ca rt e r e t t e, E. C. (1991). Perceptual scaling of simultaneous wind instrument timbres. Music Perception, 8, Ker g o m a r d, J. (1991). Le timbre des instruments à ance [Te timbre of reed instruments]. In C. Bourgois (Ed.), Le timbre, métapore pour la composition [Te timbre, metapor for te composition] (pp ). Paris: I.R.C.A.M. Kri m p o f f, J., McAd a m s, S., & Wi n s b e r g, S. (1994). Caractérisation du timbre des sons complexes, II Analyses acoustiques et quantification psycopysique [Caracterization of complex sounds timbre, II Acoustical analyses and psycopysical quantification]. Journal de Pysique IV, Colloque C5, 4, Kr u m a n s l, C. L. (1989). Wy is musical timbre so ard to understand? In S. Nielzén & O. Olsson (Eds.), Proceedings of te Marcus Wallenberg Symposium eld in Lund, Sweden (pp ). Amsterdam: Excerpta Medica. Mat e w s, M., Fr i b e r g, A., Be n n e t t, G., Sa p p, C., & Su n d - b e r g, J. (2003). A marriage of te director musices program and te conductor program. In R. Bresin (Ed.), Proceedings of te Stockolm Music Acoustics Conference (SMAC 03) (Vol. 1). Stockolm, Sweden. McAd a m s, S., Be au c a m p, J. W., & Me n e g u z z i, S. (1999). Discrimination of musical instrument sounds resyntesized wit simplified spectrotemporal parameters. Journal of te Acoustical Society of America, 105, McAd a m s, S., Wi n s b e r g, S., Do n n a d i e u, S., De So e t e, G., & Kr i m p o f f, J. (1995). Perceptual scaling of syntesized musical timbres: Common dimensions, specificities, and latent subject classes. Psycological Researc, 58, Rep p, B. H. (1992). Diversity and commonality in music performance: An analysis of timing microstructure in Scumann s Träumerei. Journal of te Acoustical Society of America, 92, Risset, J.-C. (1994). Quelques aspects du timbre dans la musique contemporaine [A few aspects of timbre in contemporary music]. In A. Zenatti (Ed.), Psycologie de la musique [Psycology of music] (1st ed., pp ). Paris: Presses Universitaires de France. Risset, J.-C., & We s s e l, D. L. (1999). Exploration of timbre by analysis and syntesis. In D. Deutsc (Ed.), Psycology of music (2nd ed., pp ). San Diego, CA: Academic Press. Sc o l e s, P. A. (1960). Te Oxford companion to music (2nd ed.). Oxford, UK: Oxford University Press. Seasore, C. E. (1938/1967). Psycology of music. New York: McGraw-Hill. (Reprinted 1967, New York: Dover Publications). Sie g e l, S., & Jo n Ca s t e l l a n Jr., N. (1988). Non parametric statistics for te beavioral sciences (2nd ed., pp. 272). New York: McGraw-Hill International Editions. Tob u d i c, A., & Wi d m e r, G. (2003). Playing Mozart prase by prase. In K. D. Asley & D. G. Bridge (Eds.), Lecture Notes in Computer Science (LNCS), 5t International Conference on Case-Based Reasoning (ICCBR) (Vol. 2689, pp ). Berlin Heidelberg: Springer-Verlag.

14 278 Matieu Bartet, Pilippe Depalle, Ricard Kronland-Martinet, & Sølvi Ystad Tod d, N. P. M. (1992). Te dynamics of dynamics: A model of musical expression. Journal of te Acoustical Society of America, 91, Tod d, N. P. M. (1995). Te kinematics of musical expression. Journal of te Acoustical Society of America, 97, Trau b e, C. (2004). An interdisciplinary study of te timbre of te classical guitar. Unpublised doctoral dissertation, McGill University, Montreal, Canada. Wes s e l, D. L. (1979). Timbre space as a musical control structure. Computer Music Journal, 3, Wid m e r, G., & Go e b l, W. (2004). Computational models of expressive music performance. Journal of New Music Researc, 33, Win d s o r, W. L., De s a i n, P., Pe n e l, A., & Bo r k e n t, M. (2006). A structurally guided metod for te decomposition of expression in music performance. Journal of te Acoustical Society of America, 119, Zölz e r, U. (1997). Dynamic range control. In Digital Audio Signal Processing (pp ). Cicester: Jon Wiley & Sons.