A Thesis. Submitted to the Faculty. Master of Arts DIGITAL MUSICS. Beau Sievers DARTMOUTH COLLEGE. Hanover, New Hampshire.

Transcription

1 Follow te Bouncing Ball: Music, Motion, and Emotion A Tesis Submitted to te Faculty in artial fulfillment of te requirements for te degree of Master of Arts in DIGITAL MUSICS by Beau Sievers DARTMOUTH COLLEGE Hanover, New Hamsire May 30, 2010 (cair) Micael Casey Talia Weatley Brian W. Pogue, P.D. Dean of Graduate Studies Larry Polansky

2 Coyrigt 2010 by Beau Sievers All rigts reserved.

3 Abstract We suggest musical emotion oerates mimetically wit resect to emotionsignifying movement. In service of tis yotesis, we resent an exeriment in wic subjects use a comuter rogram to create reresentations of several emotions in eiter music or animated motion, and te dynamic roerties of tese reresentations are quantitatively comared. We sow te reresentations created by subjects are significantly more similar witin emotion grous tan witin task grous, roviding strong evidence in suort of our yotesis. Along te way, we roose a scema for classifying cross-modal maings, and argue tat cross-modal maing in general, and music-motion-emotion maings in articular, are good candidates for universal uman redisositions. ii

4 Acknowledgements Tanks to my arents, William and Daria, for teir unconditional suort of everyting I do, no matter ow obscure, and to Joanne Ceung, for being endlessly encouraging and exceedingly atient. To my advisors: Talia Weatley, Larry Polansky, and Micael Casey; witout teir insiration and assistance, none of tis work would ave been ossible. Secial lucky-bonus tanks to Newton Armstrong, wo rovided invaluable council. To Jon Aleton, founder of te Dartmout Digital Musics rogram, for is el and advice. To Daniel Leoold, James Huges, and Swaroo Guntualli for eling wit te beavioral and fmri exeriments. To Rebecca Fawcett, for keeing everyting in order. To everybody wo talked wit me about te work, muc of wic is insired by conversations wit Walter Sinnott-Armstrong, Teodore Levin, Peter Tse, Yune- Sang Lee, David Dunn, Cristine Looser, Cris Peck, Guy Madison, Ioana Citoran, Sencer Toel, Patrick Barter, Micael Cinen, Kristina Wolfe, Paul Osetinsky, Dave Yoss, Nicolai Bur, and Brendan Landis. To my rofessors at Dartmout: Kui Dong, Carles Dodge, Doug Perkins, Aden Evens, Amy Allen, Jody Diamond, Fred Haas, Don Glasgo, Bill Kelley, Scot Drysdale, and Afra Zomorodian. And to Bronwen Evans, for awakening my interests in cognitive neuroscience and ilosoy of mind. iii

5 Contents Abstract ii Acknowledgements iii Contents iv List of Tables vi List of Figures vii 1 Introduction 1 2 Background Wat is emotion? Emotion recognition Emotion recognition in music Emotion recognition in seec Emotion recognition in uman movement Emotion as dynamic contour Cross-modal connections Infants Cross-cultural emotion Moving ast naïve universalism and relativism Evidence for cross-cultural music-emotion maings Were do we go from ere? 31 3 A beavioral exeriment A notable metodological recedent Parameterization and emotion coices Te model Te maings Music Motion 42 iv

6 3.5 Exerimental metod Results Multi-way ANOVA/GLM Analyses by emotion class Angry Hay Peaceful Sad Scared Wole dataset analyses Similarity analysis/ierarcical clustering Discussion and furter directions Testing for cross-cultural validity Imlications of te exerimental aradigm 76 4 Bibliogray 79 v

7 List of Tables 1 Summary of results from Juslin & Laukka (2004) 8 2 Angry: means and standard deviations 49 3 Angry: correlations wit magnitude greater tan Angry: LDA results 50 5 Hay: means and standard deviations 51 6 Hay: correlations wit magnitude greater tan Hay: LDA results 52 8 Peaceful: means and standard deviations 54 9 Peaceful: correlations wit magnitude greater tan Peaceful: LDA results Sad: means and standard deviations Sad: correlations wit magnitude greater tan Sad: LDA results Scared: means and standard deviations Scared: correlations wit magnitude greater tan Scared: LDA results Wole dataset: correlations wit magnitude greater tan Witin class covariance for ay and eaceful Music versus motion LDA 65 vi

8 List of Figures 1 Asects of te Egg 43 2 Screensot of te user interface 46 3 All oints on te consonance-udown lane 61 4 Hay and eaceful data oints on te consonance-udown lane 62 5 Hay and eaceful data oints on te bigsmall-bpm lane 64 6 Raw distance matrix 66 7 Regularized distance matrix 67 8 Dendrogram from regularized distance matrix 69 vii

9 1 Introduction We erceive te world all at once. Making sense of our environment requires te integration of multile simultaneous ercetual streams. Tracking relationsis between tese streams is ow we understand wat is aening around us. Tis is a matter of great evolutionary imortance: we ave good reason to be alert if wat we are looking at jums and makes a sound, or if we ear someting at or beyond te limits of our field of vision. Corresondences suc as tese are not rationally identified solemnly contemlating te roar leads into te lion's mout but are a function of involuntary ercetual rocesses. Co-occurrence, for examle, is one of several natural indicators of corresondence: wen te dynamic contours of a sound and a movement are syncronized, we tend to erceive te two as a unified wole, even if te movement is not te source of te sound. Tis tendency is flexible and romiscuous: we often ear sound as signifying movement even if tere is no real movement at all. Te caacity to cross-modally ma sound to imlied movement eled our ancestors survive in te wilderness; a more common use is te enjoyment of a well-made film. We argue tat tis caacity is wat allows us to ear music as signifying emotion. Following Meyer's (1956) yotesis tat music can be eard as imitating te dynamics of beavior, we suggest musical emotion oerates mimetically wit resect to emotion-signifying movement. Te resent work as two goals. First, we oe to rovide a reliminary teoretical framework for understanding cross-modal maing, and to oint toward ossible rograms for future researc. To tis end, we undertake an extensive review of te relevant literature. Tis review serves to draw connections between rojects wic ave someting to say about cross-modal 1

10 maing, but ave remained more-or-less isolated. In doing so, we veer far afield of emotion recognition in its most basic sense, exloring suc subjects as synestesia, infant language learning strategies, and te istory of universalist versus relativist aroaces to musicological ractice. In te course of tis review, we make two material offerings. We roose a scema for classifying cross-modal maings, and we rovide evidence tat cross-modal maing in general, and music-motion-emotion maings in articular, are good candidates for universal uman redisositions. Our second goal is to rovide emirical evidence tat musical emotion is imitative of emotion-signifying movement. To tis end, we resent an exeriment in wic subjects use a comuter rogram to create reresentations of several emotions in bot music and animated motion, and te dynamic roerties of tese reresentations are quantitatively comared. 2 Background 2.1 Wat is emotion? Any study concerning emotion is obliged to rovide at least a cursory, working exlanation of wat emotion migt be. Te question Wat is emotion? is ancient, and most or all roosed definitions seem inadequate eiter too loose for inclusion in scientific discourse, or so rigid as to sever every link to ordinary language. Taxonomizing and cataloging te istory of tis roblem is beyond te scoe of tis work, as is any serious attemt to solve it. Rater tan eroically roosing some new gymnastic definitional criteria, we dodge te roblem by slitting it in two. Directly following te work of Andrea Scarantino (2005), we see te question Wat is emotion? as concealing two related but fundamentally 2

11 searate roblems. Many (if not most) attemts to define emotion ave been stymied by a tendency to conflate tese two roblems; our strategy will be to focus on one and delegate te oter to ambitious ilosoers. Te first roblem is finding a definition of emotion suitable for scientific use. Imlied ere is te rocess of examining te inexact, scientifically unsuitable, ordinary use of te term emotion and roceeding to exlicate, a la Carna (1950), new related teoretical constructs. Tese constructs would be muc narrower in meaning tan tose imlied by ordinary language, but may outline categories constituting wat Quine (1969) calls natural kinds, or grous of tings about wic scientific generalizations are ossible. Scarantino (2005) refers to te roduction of tese constructs as te Exlicating Emotion Project. Te second roblem is figuring out ow te ordinary language term emotion is used. Rater tan constructing scientific categories wit necessary and sufficient conditions for inclusion, analysis of ordinary language suggests using a family resemblance model (Wittgenstein, 1953). Tat is, someting can be considered a member of te emotion family if it as a certain number of emotional traits, but it is ossible for family members to ossess nonoverlaing sets of traits and tus be fundamentally dissimilar. An account of emotion in tese terms would be comletely comatible wit (and indeed derived from) ordinary language, but resistant to scientific generalization. Emotions in tis sense may not form a natural kind (Griffits, 2004). Scarantino calls te task of exlaining emotional family resemblance te Folk-Emotion Project. In its understanding of emotion as a general concet, te resent work takes te Folk-Emotion Project as its basis. Investigation of ow anger or any oter emotional concet manifests can roceed erfectly well witout a recise, scientific definition. It is sufficient for most uroses to ensure tat all exemlars of te invoked concet 3

12 are emirically certified as autentic. Tat is, if subjects in an exeriment view or listen to a stimulus and label it as angry, and tis labeling is sown to be statistically significant, we sould trust tem. In oter words, at least wit resect to natural language and everyday uman exerience, emotion is watever eole say it is. Te details of an exerimental metodology may imly a certain teoretical stance toward emotion. For examle, asking subjects to rank te emotion in a iece of music on a scale from ositive to negative assumes tat emotion varies on some dimension wic corresonds to tose terms. Our metodology assumes only tat emotions can be groued into categories based on similarity. We oe tis assumtion is relatively teory-neutral; tat is, it may mes fruitfully wit dimensional, social, or oter teories of emotion, so long as tose teories admit emotions may be comared to oter emotions on te basis of similarity or dissimilarity. Tat tis is true wit resect to everyday emotional tinking in Western subjects as been confirmed by Saver et al. (1987). Saver asked subjects to sort cards rinted wit emotion words into categories, creating an emotional distance matrix wic was subjected to cluster analysis. Tis analysis found emotions fit into five broad categories: love, joy/ surrise, anger, sadness, and fear. Te resent study focuses on recognition, and not induction, of emotion. Emotion induction is a comlicated rocess not cleanly reducible to a small set of investigable factors. It is easy to conceive, for examle, of a ay-seeming iece of music wic makes a listener feel sad by way of an ironic contextual resentation, suc as in a film wen a caracter as just died. Furter, te idea tat emotional induction during music listening can occur at all is somewat controversial. Some ilosoers of music, suc as Kivy (1989) and Meyer (1956), believe listeners abitually confuse induction and recognition. Listeners 4

13 may say tey are feeling emotions symatetically wit some music, wen in fact tey are only recognizing tose emotions as being exressed. Determining te fact of te matter is difficult, in no small art because te roblem itself calls into question te judgment of te exerimental subjects. A number of strategies ave been devised to work around tis stumbling block, including observing cognitive canges during listening (Martin & Meta 1997; Balc et al, 1999), measuring ysiological arousal (Bartlett, 1996; Krumansl 1997), and taking detailed subject reorts (Kenealy 1988; Zentner et al, 2000; Sloboda & O'Neill, 2001). A review of over 100 studies by Juslin and Laukka (2004) concludes tere is sufficient evidence to claim tat music induces emotions, but also notes tat te range of emotions induced by music seems to be very different from te range of emotions wic can be recognized in it. Te reduced range of musically induced emotion is demonstrated by Sloboda and O'Neill (2001). Using an exerimental aradigm insired by Csikszentmialyi and Lefevre (1989), te autors equied subjects wit agers and aged tem at random once witin every 2-our interval during te day. Wen aged, subjects were instructed to write down in a log book information about teir most recent exerience of music listening. Subjects reorted music eiter made tem feel more ositive, more alert, and more focused in te resent, or caused tem to become nostalgic, tinking about tings and eole not resent. Te difference in range between tese reorts and everyday tinking about musically induced emotion is good evidence tat induction and recognition are decouled. Conveniently, emotion recognition in music is muc easier to study and substantiate tan induction. If a subject claims to recognize an emotion in music, tere is little reason to distrust tem, so subject reorts become muc more 5

14 useful. Furter, wile verifiable reorts of emotional induction are ambiguous and sarse, reorts of emotion recognition are secific and lentiful. 2.2 Emotion Recognition A survey of te literature on recognition of emotions in various modalities follows, wit secial emasis on results wic suggest recognition in different modalities may be governed by similar rocesses Emotion recognition in music Tat emotion may be recognized in music is uncontroversial. More interesting is to ask wic emotions may be recognized, and wic are off-limits. Certainly some music is ay, and oter music is sad, but is tere jealous music? Guilty music? Furter, wat roerties allow us to distinguis ay from sad music? Are tese judgments unreliable and subjective, or is tere a broad consensus about wat music is ay and wat music is sad? Juslin and Laukka's (2004) meta-study concluded tat music listeners' judgments regarding emotion are systemic and reliable, and can tus be redicted wit reasonable accuracy. Suc judgments were only marginally affected by te level of musical training, age, and gender of te listener. Tat is, tere is broad consensus regarding te recognition of emotional content in music. Furter, te emotion recognition rocess is nearly immediate. Peretz et al. (1998) found subjects were able to accurately judge te emotional tone of musical excerts in well under 3 seconds. However, subjects were less likely to agree on differences witin emotional categories, indicating limits on te recision wit wic music can reresent emotion. Juslin (2005) lists tree ossible reasons for tis imrecision. First, it may be tat music's ability to 6

15 communicate emotions is eavily deendent on its similarity to oter forms of non-verbal communication and is tus similarly limited. Second, tere is a great deal of redundancy built in to te structure of musical features wic communicate emotion, lacing a limit on exressive secificity. Tird, music is not designed solely to convey emotion, and so oter considerations may often take recedence. Juslin and Laukka (2004) also comiled a list of te emotional states most frequently recognized by listeners along wit teir attendant musical features. Teir aroac imlies a arameterization of music based on te terminology of Western music teory. Teir results are summarized in table 1. 1 Wile tis analysis is limited by its reliance on te vocabulary of Western music teory, it does indicate tat emotion recognition is reliably accomanied by certain sets of musical features. Tat is, te resence of tese musical features seems to be a sufficient condition for recognition of te emotion. Juslin (2000) suggests a robabilistic model for quantifying te contributions of various musical features to bot te roduction and recognition of emotion in music. Tis model is based on Brunswik's (1956) lens model and Hursc's (1964) lens model equation (LME). Tirty listeners judged te emotions exressed in erformances of tree sort musical ieces by tree rofessional guitarists. Multile regression analysis was used to determine te relationsi between musical features and erformer intention, as well as te relationsi between musical features and listener interretation. Te LME was ten alied to quantify ow closely te exressive codes of te erformers matced te interretive codes of te listeners. Tis aroac yielded results similar to tose summarized above: Anger was associated wit fast temo, ig 1 Wile timing variability is associated wit rubato, Juslin & Laukka are unclear about ow timing variability is distinguised from temo variability. 7

16 Anger Hainess Tenderness Sadness Fear Fast temo, small temo variability, minor mode, atonality, dissonance, ig sound level, small loudness variability, ig itc, small itc variability, ascending itc, major 7t and augmented 4t intervals, raised singer s formant, staccato articulation, moderate articulation variability, comlex rytm, sudden rytmic canges (e.g., syncoations), sar timbre, sectral noise, fast tone attacks/decays, small timing variability, accents on tonally unstable notes, sar contrasts between long and sort notes, accelerando, medium-fast vibrato rate, large vibrato extent, micro-structural irregularity Fast temo, small temo variability, major mode, simle and consonant armony, medium-ig sound level, small sound level variability, ig itc, muc itc variability, wide itc range, ascending itc, erfect 4t and 5t intervals, rising micro intonation, raised singer s formant, staccato articulation, large articulation variability, smoot and fluent rytm, brigt timbre, fast tone attacks, small timing varibility, sar contrasts between long and sort notes, medium-fast vibrato rate, medium vibrato extent, micro-structural regularity Slow temo, major mode, consonance, medium-low sound level, small sound level variability, low itc, fairly narrow itc range, lowered singer s formant, legato articulation, small articulation variability, slow tone attacks, soft timbre, moderate timing variability, soft contrasts between long and sort notes, accents on tonally stable notes, medium fast vibrato, small vibrato extent, micro-structural regularity Slow temo, minor mode, dissonance, low sound level, moderate sound level variability, low itc, narrow itc range, descending itc, flat (or falling) intonation, small intervals (e.g., minor 2nd), lowered singer s formant, legato articulation, small articulation variability, dull timbre, slow tone attacks, large timing variability (e.g., rubato), soft contrasts between long and sort notes, auses, slow vibrato, small vibrato extent, ritardando, micro-structural irregularity Fast temo, large temo variability, minor mode, dissonance, low sound level, large sound level variability, raid canges in sound level, ig itc, ascending itc, wide itc range, large itc contrasts, staccato articulation, large articulation variability, jerky rytms, soft timbre, very large timing variability, auses, soft tone attacks, fast vibrato rate, small vibrato extent, micro-structural irregularity Table 1: Summary of results from Juslin & Laukka (2004) 8

17 sound level, a lot of HF [ig frequency] energy in te sectrum, legato articulation, and small articulation variability; sadness was associated wit slow temo, low sound level, little HF energy in te sectrum, legato articulation, and small articulation variability; ainess was associated wit fast temo, ig sound level, intermediate amount of HF energy in te sectrum, staccato articulation, and muc articulation variability; fear was associated wit slow temo, very low sound level, little HF energy in te sectrum, staccato articulation, and large articulation variability. (Juslin, 2000) Exressive codes used differed from erformer to erformer, but desite tis, eac erformer was intelligible to all of te listeners. Presumably because melody is difficult to arameterize, and erformers are scarce, te melodies used were fixed trougout te exeriment, and melody-related arameters were not included in te regression analysis. Tis limitation revented te study from assessing te effects of canges in contour, ste size, consonance, etc. Wit resect to emotional exerience in day-to-day life, it is imortant to note tat te emotions identified by te above studies (ainess, sadness, anger, fear, and tenderness) are a very small subset of states tyically considered emotional. Saver et al. (1987) asked 100 subjects to rate 213 ossible emotion names on a scale of 1 to 4, wit 1 meaning I would definitely not call tis an emotion and 4 meaning I would definitely call tis an emotion. A number of ig ranking candidates are notably absent from most studies on music and emotion, including jealousy (3.81), grief (3.65), guilt (3.53), embarrassment (3.49), same (3.43), and disgust (3.42). Wat emotions can and cannot be exressed in music may be valuable information wit resect to ow, exactly, te recognition of musical emotions takes lace. 9

18 2.2.2 Emotion recognition in seec A comlete review of istorical aroaces to emotional communication in seec is beyond te scoe of te resent work. I offer a brief survey of recent emirical work, drawing eavily from Scerer (2003). Scerer (2003) notes tat te basis of any functionally valid communication of emotion via vocal exression is tat different tyes of emotion are actually caracterized by unique atterns or configurations of acoustic cues. [...] Witout suc distinguisable acoustic atterns for different emotions, te nature of te underlying seaker state could not be communicated reliably. Additionally, canges in acoustic arameters are linked to ysiological canges, i.e. exeriencing a given emotion canges te way eole seak. Te following quote is reresentative of ow tis idea is tyically framed in te literature: For instance, many of us ave exerienced talking in an unwittingly loud voice wen feeling gleeful, seaking in an uncaracteristically ig-itced voice wen greeting a sexually desirable erson, or talking wit marked vocal tremor wile giving a ublic seec. (Bacorowski, 1999) It is imortant to note tat tis is not always true. Deliberate, exressive modulation of acoustic arameters not related to some autentically exerienced emotional state is certainly ossible, as in te cases of acting and decetion. (Ekman et al., 1976; Anolli et al., 1997) Te comosition of music offers an analogous circumstance; a work may exress or convey emotions te comoser did not feel during comosition, nor te erformer during exibition. Studies reviewed by Scerer (2003) of ow emotion is encoded into seec examined recordings of eole in emotionally trying situations, subjects under te influence of emotion-altering sycoactive drugs, subjects wo ad undergone a battery of laboratory rocedures designed to induce emotion, and actors simulating emotional states. Findings across tese studies were consistent, 10

19 sowing a stable set of acoustic features associated wit eac emotion studied. For examle, anger was associated wit increases in intensity, seed, te mean value of formant 0 (F0), and descending sentence contours; fear wit increases in intensity, seed, indeterminate F0 range, and indeterminate contour; sadness wit decreased intensity, seed, F0 mean and range, and falling contours; joy wit increased intensity, seed, and F0 mean and range. It is significant tat all of tese acoustic arameters ave equivalents in music Emotion recognition in uman movement Like music and seec, bodily movement can act as a conduit for te exression of emotion. Te standard ractice for cature and analysis of uman movement in isolation is te oint-ligt model, were small ligts are attaced to te joints of actors wo are filmed moving in a room (Joansson, 1973). Viewing te ligts, and not te body attaced to tem, rovides a way to observe te movement of a erson isolated from anyting wic could comlicate interretation, suc as facial exression. Wile viewing oint-ligt movement, eole are able to identify suc attributes as gender (Kozlowski and Cutting, 1977), vulnerability (Gunns et al., 2002), and emotion (Atkinson et al., 2004; Makeig, 2001; Pollick et al., 2001). Emotion may be erceived in oint-ligt motion even wen emotional exression is not te rimary goal of te actor. (Pollick et al., 2001) Unlike researc into music and seec, oint-ligt motion studies tend not to address te relationsi between arameterization of te stimuli and subjects' categorical judgments. Tis is robably because comlex oint-ligt motion does not ave any intuitive or obvious arameterization, leading most researcers to favor macine learning analyses. Tese analyses yield dimensions on wic automated classification of emotional movement is ossible, but wic do not necessarily ave any relationsi to uman ercetion. Below, we focus on 11

20 studies wic eiter take uman ercetion as a starting oint, or reframe te results of macine learning analyses in terms of ercetually valid categories. Castellano et al. (2007) used automated classification tecniques to analyze video of uman movement wit resect to emotion. Tey arameterized movement in terms of a number of roerties related to amlitude variation and sectral centroid (a weigted average level of energy), and arameterized emotion along te dimensions of valence and arousal. Wile moderately successful, teir model confused negative emotions wit ositive emotions aving similar arousal caracteristics (e.g. angry and ay), and confused ositive emotions wit oter ositive emotions wit oosite arousal caracteristics (e.g. ay and eaceful). Amaya et al. (1996) created a system for modifying catured neutral uman movement suc tat it exressed various emotions. Teir system focused on amlitude variation, corresonding wit Pollick's (2001) activation dimension, but tey suggest tat oter emotional canges in te movement (eras variation in Pollick's leasantness dimension) may be related to ase relations between joints. Badler et al. (1999) suggest a arameterization based on Laban Movement Analysis (Laban, 1960) ossibly a very interesting direction but do not quantify teir model or demonstrate ow it migt be used. Pollick et al. (2001), using a dimensional model of emotion based on activation and leasantness, found ig-activity emotions were associated wit greater velocity, acceleration, and jerk in te exressive movement. Bernardt and Robinson (2007) suggest a similar model. Interestingly, Pollick et al. (2001) found teir results eld wit resect to te activation dimension even wen te oint-ligt dislays were scrambled so tey were no longer consistent wit uman movement. Tis suggests certain dynamic features resent in oint- 12

21 ligt motion are sufficient for emotion recognition even witout a visible, moving uman body. 2.3 Emotion as Dynamic Contour In te modalities discussed above, eac recognizable emotion is associated wit a set of unique features. In music, for examle, ainess is associated wit fast temo, major mode, ascending itc, and so on. Remarkably, tere are similarities in arametric variation across modalities for eac emotion. Sad music, for examle, as a slow temo, low and descending itc, and is dissonant (Juslin and Laukka, 2003). Sad seec as a slow articulation rate, low fundamental frequency, descending itc, and is dissonant (Scerer, 2003). Sad door-knocking movement is slow and slack (Bernardt and Robinson, 2007) and associated wit low velocity, acceleration, and jerk (Pollick et al., 2001). Temo, articulation rate, and slowness are all variations on te teme of seed, and sadness is slow weter in music, seec, or movement. Tis suggests te signifying ower of a given arameter isn't limited to one medium, but can cut across media and ercetual modalities. Indeed, at least for seec and music, tis tendency toward cross-modal arametric similarity as been confirmed by Juslin and Laukka (2003), a meta-study of 104 aers wic concludes tat music erformance involves mainly te same emotion-secific atterns of acoustic cues as does vocal exression. Moving forward, we review a number of studies wic consider in more detail maings wic act across ercetual modalities. We use te term crossmodal maing to refer to any reliable association of activity in one modality to activity or imlied activity in anoter. We suggest cross-modal maings may be classified using te following tree-level scema. First, a maing may be eiter ercetual or cognitive. A cognitive maing requires conscious effort to 13

22 understand; an examle is looking at a cart and evaluating its meaning. In contrast, ercetual maings are automatic, obligatory and unconscious: wen we ear a fast, intense series of footstes, we know immediately tat tere is a erson running, witout engaging in any active reading rocess. Second, maings may be eiter intuitive or learned. Intuitive maings are resent from birt and require no acculturation or study. A good candidate for an intuitive maing between movement and music in infants is described by Pillis-Silver and Trainer (2007) (discussed in detail in te following section). Learned maings require study or acculturation; a familiar examle of a learned maing is te relationsi between words in a language and teir meanings. Provisionally, we suggest a tird level. Maings may be eiter isomoric, analogical, or arbitrary. A maing is isomoric wen te source and te result arameters occuy te same reresentational sace. For examle, slow music maing to slow movement. We refer to tis kind of maing as cross-modal arametric isomorism. Conversely, a maing is analogical wen te source and result arameters occuy different reresentational saces wic are maed to one anoter, usually in terms of intensity or magnitude. For examle, visual brigtness maing to itc eigt. We call tis tye of maing inter-arametric analogy. Finally, enomena maed arbitrarily may ave no reresentational similarity, and are tyically associated by rote memorization. Altoug te tree levels of tis scema are concetually indeendent, tere are correlations between levels; for examle, arbitrary maings are always learned. We suggest tat to recognize emotion in music, seec, or movement is, in art, to comare te dynamic contour of a stimulus via cross-modal maing to an internal dynamic model of emotion. Te cross-modal maings imlicated in tis rocess may ave any of te six roerties we suggest in our classification scema; tey may be ercetual or cognitive, intuitive or learned, and 14

23 isomoric or analogical. In articular, we would like to igligt te ossible imortance of automatic ercetual and intuitive maings in te understanding of emotion, esecially insofar as tat understanding aears to be a uman universal (tis is discussed at lengt in section 2.6). 2.4 Cross-modal connections How does cross-modal maing come to be ossible? Wat are its uses, tendencies, and limits? Te following section is a survey of researc wic, rater tan focusing on a articular medium or modality, directly investigates cross-modal maing itself. Film and television viewing is a commonlace scenario were te crossmodal influence of music lays an imortant role. Coen (1993) found tat subjects' judgments of te affect of a bouncing ball were modified by music in a rougly additive manner. Tat is, wen subjects viewed a aily bouncing ball, laying ay music made it aear more ay, and laying sad music made it aear less ay. Coen (1993) also found tat wen te music contained ascending and descending major triads, tis increased erceived ainess, but wen minor triads were layed, subjects' judgments were indeterminate. Coen (1993) also describes two furter exeriments wic used more realistic musical and visual stimuli and acieved similar results. Eitan and Granot (2006) asked 95 college students to, wile listening to music, visualize internally and ten describe using a forced-coice questionnaire te movement of an imaginary animated caracter. Te musical ieces used were sort melodies, designed in airs suc tat one iece featured an increase in some musical arameter and te oter featured a decrease, wit oter arameters eld constant. Parameters varied included dynamics, itc direction, itc intervals, attack rate or inter-onset interval, motivic ace, and articulation. Tey found 15

24 canges in te music affected imagined motion significantly and diversely as subjects emloyed a variety of music-to-motion maing strategies. Te most statistically significant results include increases in volume maing to an aroac motion or an increase in seed, decreasing volume maing to descending motion or movement away from te subject, ascending itc to ascending motion, descending itc to descending motion, and decreasing interonset intervals (accelerandi) maing increases in seed and vice-versa. Eitan and Granot note tat a number of te maings used by subjects were analogical in nature, suc as ascending itc eigt maing to ascending osition in imaginary sace. Furter, some cross-modal maings were directionally asymmetrical, wit musical arameter canges aving a greater effect on imagined motion in one direction tan te oter; e.g. falling itc maed very strongly to descending motion, but te relationsi of rising itc to ascending motion was weaker. All results were largely invariant wit resect to te level of subjects' musical training, altoug subjects wit training tended to aly maing strategies more consistently. An earlier study by Eitan and Granot (2003) also found inter-arametric analogies were imortant for determinations of stimulus similarity in music. Subjects judged musical rases wit similar arametric contours as similar to one anoter, even if tose contours were alied to different arameters in eac musical rase. For examle, rases wit accelerating temo were judged similar to rases wit increasing itc. Tis demonstrates subjects are able to erceive arametric contours as distinct from musical rases as gestalts. Pillis-Silver and Trainer (2007) demonstrated te effect of bodily movement on rytm ercetion, sowing tat, as tey ut it, ow we move will influence wat we ear. Tey layed rytmically ambiguous musical rases to infant subjects aged 7-monts wile gently bouncing te subjects in a rytm 16

25 tat imlied eiter a marc or a waltz. In a listening test, were te musical rases were rytmically disambiguated by te addition of accents in eiter dule or trile meter, te subjects listened significantly longer to te rytm tat matced teir movement. Wile teir results didn't deend on visual stimulation, in teir introduction tey draw an interesting analogy between rytmically and visually ambiguous figures. Tey comare teir marc-waltz rases, wic can be eard eiter in dule or trile meter, to Rubin's (1915) face-vase image, wic can be seen as eiter a vase or two faces looking at one anoter. Saenz and Koc (2008) resented evidence of ercetual (i.e. automatic and imenetrable) maings in visual-audio synestetes, a class of subjects wo involuntarily exerience sound sensations alongside visual canges suc as flasing ligts and fast movements. Teir exeriment exloited a well-known cross-modal asymmetry: normal eole are retty good at identifying auditory rytms and evaluating teir similarity, and retty bad at accomlising te same wit visual rytms. To confirm reorts of visual-audio synestesia, Saenz and Koc layed airs of sort rytms to two grous of subjects: one normal grou, and te oter a grou of synestetes wo claimed to ear visual flases as auditory bees. Half of te rytm airs were resented as audio, and te oter alf as visual flases. Subjects were asked to evaluate weter te two rytms in eac air were te same or different. Normal subjects erformed well on te auditory task, but not te visual. Te synestetes, wo claimed to ear visual flases as auditory bees, erformed well in bot domains. Interestingly, over te course of te exeriment, te synestetes reorted te synestetic sounds tey eard along wit te visuals canged to matc te real sounds layed during te auditory tests. 17

26 Synestetic connections suc as te above are not mere flukes or irregularities, unredictable variations from a cross-modally segregated norm, but in fact are widesread and tougt to undergird te ercetual rocesses of normal individuals. In teir excellent review of recent researc on synestesia, Sector and Maurer (2009) suggest two ossible develomental causes of synestetic ercetion. According to te cross-activation teory, synestesia is te result of incomlete runing of synatic connections between adjacent brain areas. Te disinibited feedback teory suggests synestesia is caused by reentrant feedback from iger cortical areas failing to inibit te effects of connections between rimary sensory cortical areas. Sector and Maurer state tat bot of tese teories redict synestesia would be ubiquitous among normal individuals in early cildood and would ersist to some extent in normal adults. Grous of synestetes (e.g. visual-audio, word-color, etc) tend to exibit te same cross-modal maings. Some of tese maings are based uon interarametric analogy. For examle, synestetic adults wo erceive auditory itc as visual color tend to ma itc eigt to brigtness, wit iger itces resulting in brigter colors (Sector and Maurer, 2009). Tis cross-modal interaction is sufficiently strong tat teir ability to discriminate itc is affected by te luminosity of te earing environment. In itc identification tests wit a ligt used as a distractor, synestetic subjects resonded slower and less accurately if te distractor was oosed to te synestetic ercet of te itc; i.e. sining a ligt at a synestete wit colored earing interferes wit teir ability to ear correctly. (Marks, 1987, as cited by Sector and Maurer, 2009) Te maing of iger itc to greater brigtness also olds to some extent in normal adults and toddlers, suggesting tat tere may be a natural 18

27 maing (wat we would classify as an intuitive, ercetual maing) between itc eigt and brigtness. Sound and sae are also associated. Sector and Maurer (2009) note tat sar visual saes go wit words tat roduce a small, constricted movement of te tongue and mout (e.g., sike, oint). Tis is exemlified by an exeriment were cildren and adults were asked to matc a nonsense word (e.g. takete, kiki, maluma, bauba) to a 2-dimensional sae. Words like takete and kiki were reliably matced wit jagged, siky saes, wile words suc as maluma and bauba were matced to more rounded, bulbous saes (Köler, 1929). Sector and Maurer (2009) elaborated on tese exeriments, testing for sound-sae maings in toddlers using a wide variety of sounds and saes, finding te association between non-rounded vowels and jagged saes, and rounded vowels and rounded saes was consistent. Tey also determined te effect occurred early enoug in develoment to influence te learning of language. Furter, te association of rounded sounds and rounded forms, and sar sounds and angular forms seems to old across cultures. Te takete/ maluma exeriment was erformed on 14 year-old cildren wo soke no Englis, but Swaili and te Bantu dialect of Kitongwe, wit similar results to Englis seaking subjects (Davis, 1969). Furter, tere aear to be sae corresondences between real, non-nonsense words wic old across language barriers. Koriat and Levy (1979) sow tat Hebrew seaking adults could matc Cinese caracters wit teir corresonding Hebrew word wit better-tancance accuracy, and Berlin (1994) sowed tat Englis seakers were able to accurately sort Huambison words based on weter tey referred to a bird or a fis. Ramacandran and Hubbard (2001) seculate tat tese enomena arise from connections between contiguous cortical areas mediating decoding of te visual ercet of te nonsense sae (round or angular), te aearance of te 19

28 seaker s lis (oen and round or wide and narrow), and te feeling of saying te same words oneself, invoking te idea of a natural maing. Sector and Maurer (2009) continue tis train of tougt, taking te cross-language results to suggest tat tis natural sound-sae maing as significant influence on cross-cultural evolution of language. 2 Sector and Maurer (2009), in a section entitled A Common Code for Magnitude, note tat a great many synestetic effects can be exlained in terms of cross-modal arametric isomorism or arametric analogy. Tey suggest a natural redisosition to ma magnitude cross-modally exists from birt, and osit a likely evolutionary exlanation: it would leave more energy for te learning and working out of arbitrary maings not related to magnitude, wic tend to be individually meaningful. Teir examle of an individually meaningful maing is from te timbre of Mom's voice to er face. 2.5 Infants Treub (2001) sows infants are caable of and redisosed toward te recognition of melodic and rytmic contours as distinct from melodic gestalts. Infants are able to recognize melodies after transosition, and rytms after rases ave been sed u or slowed down, so long as te relative lengts of te notes remain te same. Treub identifies melodic contour as te most salient musical feature for infant listeners, and references studies (Fernald, 1991; 2 Sector and Maurer (2009) suggest tat in te case of sound-sae maings like tose used in te kiki/bauba exeriment, te maing is te result of ysical symaty, were te sarness in te language is analogous to tigtness or sarness in te mout. Anoter arsimonious interretation is tat tis relationsi is not one of analogy, but ercetual isomorism: te auditory sikiness may be a ercetually salient dimension isomoric to visual sikiness. Were te word takete migt ma to an image wit a small number of sikes, takakekatakekteke migt ma to an image wit a great deal of sikes. Tis kind of sikiness may be related to te rate and amlitude contour of variation in te sectral centroid of te sound over time. 20

29 Fernald et al., 1987; Paoušek, 1992; and Lewis, 1951) wic suggest melodic contour may also be te most salient feature of moters' seec to relinguistic infants. Wen moters seak to infants, tey sli into moterese or infant-directed seec, wic is marked by increased itc and dramatically exaggerated melodic contour. Te melodic exaggeration of infant-directed seec occurs in all cultures (Treub, 2000). Treub (2001) notes tat infant-directed singing is distinguised from tyical singing styles by increased itc (altoug not quite as ig as infant-directed seec), slow temo, and slurred articulation of words. Te functions of infant-directed seec and singing seem to be to cature attention, moderate arousal, and develo te emotional bond between moter and infant. Te contour-affect relationsi imlicated in infant-directed seec is of a different kind tan te maings discussed above: tere is no evidence tat infants listening to infant-directed singing or moterese recognize tem as reresenting anyting. Because re-linguistic infants are unable to exlicitly relate a narrative of teir exeriences, it would be difficult to design an exeriment wic would rovide satisfactory evidence tat reresentation was at lay. Neverteless, tat tese modes of communication evoke affective states suggests te fundamental relationsi between contour and affect exists rior to acculturation. 3 Saffran et al. (1999 and 2008) examine te language learning rocess in infants. Tey note tat languages exemlify exactly tose structures tat umans are best able to learn, suggesting at least some asects of structure may emerge from constraints imosed by learning itself. In addition to being a ossible bedrock on wic more exlicitly reresentational contour-emotion 3 Treeb (2000) and Dissanayake (2000) suggest a number of evolutionary functions tis relationsi migt serve. 21

30 maings may develo, we conjecture tat contour-affect associations may be beneficial to te language learning rocess, and terefore may in turn broadly affect te develoment of language on a global scale. Tis would go some lengts toward exlaining some of te inter-language effects described in te revious section. Tis is also robably a two-way relationsi: listening to ambient adultdirected seaking influences te contour of affective exressions on te art of te infant. Mame et al. (2009) sows some reliminary evidence tat te contour of newborns' crying melodies are saed by te contour of teir native language. Reresentation of emotion as cross-modal dynamic contour recalls Stern's (1985) concet of vitality affects or vitality contours basically, feelings reresented by abstract 'forms' wic e suggests are imortant in early communication between moter and cild. Stern's own work leaves vitality contours vaguely defined and emirically imenetrable (Køe et al., 2008). Te resent work may to some degree assist in te exlication of vitality contours as investigable enomena. 2.6 Cross-cultural emotion Moving ast naïve universalism and relativism Wat if tis entire discourse is olluted by te cultural contingency of language and tougt? It may be tat te notion of ainess is reresented entirely differently in one culture tan anoter; indeed, all reresentation of emotion in music may be artially or wolly destabilized wen transorted across cultural borders. If tis is so, ow can it make sense to refer to emotion in music witout some geograic or cultural qualification, e.g. Western music, Hindustani music, etc.? Evidence from studies of synestesia suggests tere are certain linguistic- 22

31 formal associations wic old across cultures eras tere are musicalemotional associations wic old as well. In tis section, I discuss roblems wit wat I refer to as te naïve universalist and naïve relativist ositions on cross-cultural emotion in music, resent evidence from te literature tat tere are cross-culturally valid musical-emotional associations, and discuss ow tese associations neiter exclude genuine difference between cultures, nor imly any universal musical systems (i.e. no universal language of music). Music, as understood by wat I will call te naïve universalist osition, may be suerficially different in one culture or anoter, but as some common core wic all eole exerience te same way. Te exlanation for tis sared exerience is te sared structure of te uman body: everybody's body works more or less te same way; we all ave ears and a brain. Universal musical exeriences are te result of an interaction between te ysical roerties of a sound and te structures of our sense organs. Autors taking tis aroac tend to conflate feelings wit ercetual exeriences wit ysical enomena. If we exerience music as aving a feeling (e.g. sadness), wic seems to correlate wit a ercet (e.g. dissonance), and we ave got universal sameness on our mind, ten te natural ting to do is treat eiter te ercet, te feeling, or bot as ysical roerties of te sound. Plato's treatment of music in Te Reublic is a aradigmatic examle of tis aroac (Plato, 1992). Plato associates eac musical mode wit an emotional state, and bans from te Reublic tose modes wic evoke emotions undesirable from te ersective of government. Imlied in tis aroac are two assumtions: 1) tat certain collections of itces ave certain evocative effects, and 2) tat some of tese effects are good for culture, and some bad. Plato is undertaking an engineering roject were music (or control over musical dissonance) is a means and culture is te end. His assumtion tat music's 23

32 emotional effects are indeendent of acculturation imlies te naïve universalist osition: emotional resonses to music are determined by sonic content, not cultural context. Significantly, tis same assumtion imlies a teory of music ercetion in wic dissonance, and tereby emotional feeling, is tougt of as a ysical roerty of sound and not a subjective ercet. Many later teorists ave tried to make Plato's imlication exlicit. Leibniz famously quied tat music is te leasure te uman mind exeriences from counting witout being aware tat it is counting, and suggested in (Leibniz, 1714) tat te ercetion of dissonance was related to te subconscious calculation of frequency ratios. Wile te exact ysical correlate of dissonance varies from autor to autor, basically similar ositions are eld by Euler (1739), Stumf (1890), Helmoltz (1912), etc. A brief summary of tese autors' teories of dissonance ercetion can be found in (Lundin, 1947). All of tese teories of sound ercetion are related to cultural universalism via teir treatment of dissonance. If a cue-based model of emotion ercetion is assumed, were cues like dissonance are considered ysical roerties and not subjective ercets, ten it seems reasonable to exect musical emotion to be basically invariant across cultures. Te naïve universalist aroac is attacked by Lundin (1947) on te basis tat its account of dissonance is incoerent, and by etnomusicologists suc as Meriam (1964) and Blacking (1965) on te basis tat it fails to account for diversity and difference in te music of non-western cultures. Taking dissonance as its focus, Lundin's strategy is to drive a wedge between ercets and ysical enomena, suggesting our ercetual exeriences are culturally contingent: te way we ear is affected by te culture in wic we live. He callenges te ysical correlates of dissonance suggested by Euler, Helmoltz, and Stumf: dissonance can't simly be te subconscious calculation of frequency ratios, 24

33 were larger frequency ratios mean greater dissonance, for examle, because some out-of-tune intervals, e.g. 99:201 Hz, are still erceived as consonant. Tere must be some rocess, robably conditioned by exerience, wic allows us to erceive 99:201 Hz as close enoug for rock 'n' roll. According to Lundin, ten, dissonance is not a ysical enomenon in itself, but a ercet resulting from a discriminative reaction or subconscious judgment made on te basis of cultural exerience. Lundin's wedge between ercets and ysical roerties is not unlike te division Scarantino establises between Folk Emotion and Exlicated Emotion. According to Lundin, dissonance (like emotion) is watever eole say it is, and eole in different cultural contexts may well say different tings. Te contrasting aroaces of Leibniz et al. are attemts to exlicate a teoretical construct called dissonance wic could be scientifically useful, but may not line u wit ordinary language use. Again assuming a cue-based model of emotion ercetion, but were cues suc as dissonance are culturally conditioned judgments, one would exect to see dramatic variations in ow different cultures exress emotion in music. Naïve relativism so exressed disallows any crosstalk between folk and exlicated understandings of musical features, and any interaction between te ysical or biological and te cultural. Tis aroac is exemlified by Meriam (1964) and muc of Blacking's early work on te music of te Venda, a eole living in te Transvaal in nortern Sout Africa (e.g. Blacking, 1965). For tese autors, culture is te sureme and eras single factor constitutive of musical form: Every iece of music as its own inerent logic, as te creation of an individual reared in a articular cultural background, and in terms of tis tere is ultimately only one exlanation of its structure and meaning. (Blacking, 1973 as quoted by Agawu, 1997) For insigts into te structure of music, Blacking 25

34 looks toward relationsis wit dance, language, and social organization rater tan any sort of biological redisosition. Te relativist aroac is sketical of any claim tat music from one cultural context is similar to music from anoter. Suerficial similarities may be entirely coincidental. Wile, to Western ears, it may seem tat some culturally and geograically searate grous utilize te same musical material, te cultural contexts and organizational rinciles at lay may be entirely different. Only dee antroological study of culture can sed ligt on ow music is eard. Wile offering obvious benefits (not te least of wic is avoiding te tras of naïve universalism suggested above), te drawbacks of tis osition are substantial. If musical cultures are fundamentally incommensurable wit one anoter, wat are we to make of certain striking similarities? Witout making cross-cultural comarisons, ow are we to undertake analysis of musical cultures wic offer no internal analytic vocabulary? As Lundin (1947) okes oles in te universalist osition by undermining too-tigtly exlicated definitions of dissonance, Kolinski (1967) callenges te naïve relativist view to account for some striking emirical observations. Taking te universality of te uman vocal aaratus as a starting oint, e asks 1) wat causes te singer to select certain tones out of tis itc continuum and to organize tem into coerent structures; and 2) wy similar atterns of tonal construction can be found in widely searated areas and in strongly contrasting cultures. He goes on to suggest octave equivalence (tat is, te recognition of itces wit frequencies related by an aroximately 2:1 ratio as being members of a itc class) as a musical universal, as well as te resence of fifts and oter small frequency ratios, and categorical discrimination of itces. Evidence of te universality of tese features is offered in Kolinski (1967), Treub (2000), McDermott & Hauser (2005), and Nettl (1956, 1983). 26

35 Kolinski's (1967) aroac to tis evidence suggests a softening of bot te relativist and universalist ositions suc tat bot nature and culture are allowed influence. Tis view is afforded by te identification of very simle ercetual universals, suc as octave equivalence, as enomena on wic comlex notions suc as dissonance or musical scales must suervene. After being faced wit evidence of universality, Blacking (1995) offered te following instructive aroac: I suggest tat an accurate and comreensive descrition of a comoser's cognitive system will rovide te most fundamental and owerful exlanation of te atterns tat te music takes. 'Cognitive system' includes, of course, all cerebral activity involved in motor coordination, feelings, and cultural exeriences, as well as te comoser's social, intellectual, and musical activities. Even if we regard tem solely as 'sonic objects,' te notes of a iece of music are te roducts of cognitive rocesses. Te solution is not to clarify a transcendental boundary between biology and culture, but to acknowledge tat tey form a couled system, wit eac aving an influence on te oter. We follow tis moderated relativist aroac in our reliance on folk terminology and our grounding of terms suc as dissonance in ercetual studies instead of teory. As we understand tem, culturally conditioned ideas suc as dissonance do not refer to a single ysical enomenon, but a ackage of loosely linked roerties suc as frequency ratio caracteristics, loudness, context of resentation, timbre, and so on. Learning wat dissonance means is associating tis set of roerties wit teir roer name as situated witin a cultural context. Some of tese roerties are temselves torny, densely acked cultural terms timbre, for examle, as no obvious ysical correlate and myriad uses. Oter roerties may be basic to te uman ercetual aaratus. For examle, categorical itc ercetion, wile not being a sufficient condition 27

36 for a cultural understanding of dissonance, is robably universal, and ideas like dissonance robably deend uon it. Once te idea is learned, te collection of roerties is erceived as te identifying term, e.g. a tone at a ig volume, wit a ars timbre, a ig-numbered frequency ratio, and in a certain cultural context is erceived as a dissonance. At te same time, activation of te idea may also trigger reflection uon te interaction between te cultural context and te set of distinguisable roerties available to te sensory aaratus, wic may in turn inform or udate wat te idea means. Tis feedback loo between reflection and ercetion allows for te resence of cross-cultural universals, but acked into culturally relative terms in different combinations and degrees. It also allows for intercultural difference, as well as te sliage of meaning over time. Tis aroac as additional imlications for our lans to test our researc cross-culturally, outlined in section Evidence for cross-cultural music-emotion maings Ekman's classic studies (Ekman et al., 1969; 1971) demonstrated consistent emotion recognition in facial exressions in numerous literate and reliterate cultures, some of wic ad minimal contact wit Westerners rior to te exeriment. Tis finding was te first ositive evidence tat emotions are construed and exressed similarly across cultural boundaries. Scerer (2003) extended tis line of study beyond facial exression, finding tat vocal exressions of emotion are also recognized wit better tan cance accuracy across cultures. Tis and oter findings from Scerer et al. (2001) are interreted by te autors as evidence for te existence of universal inference rules from vocal caracteristics to secific emotions across cultures. (Scerer, 2003) Sauter et al. (2010) added to tese findings, sowing tat nonverbal emotional vocalizations were bidirectionally recognizable between Western articiants in 28

37 teir study and culturally isolated Namibian villagers. Te similarities and corresondences between musical and linguistic exressions of emotion addressed above suggest a concerted study of cross-cultural musical exression. Some reliminary stes are summarized below. Balkwill and Tomson (1999) layed selections of Kyrgyz, Hindustani, and Navajo music to Western subjects in order to comare teir assessment of te music's emotional content wit te music's cultural-emotional association. For an initial ilot exeriment, tey used a model of emotion limited to searate ratings of joy and sadness. Tey found te Western listeners assigned iger joy scores to music considered joyful in all tree traditions, and iger sad scores to music traditionally considered sad. Tey also found tat te joy rankings were ositively correlated wit te temo of te music, wile te sadness rankings were negatively correlated. Tese results were followed by a more in-det study of emotion recognition by Westerners in Hindustani music, were te emotional alette was exanded to include anger and eacefulness, and te temo, melodic comlexity, rytmic comlexity, itc range, and timbre of te music were analyzed to determine ow eac of tese musical arameters contributed to emotion recognition. Tey found tat Western listeners correctly rated te emotions of te ragas in every case excet for eace, wic was confused wit sadness. Tey found joy was correlated wit temo and melodic comlexity, sadness was correlated wit melodic comlexity, but negatively correlated wit temo, anger was correlated wit sar timbre, and eace was negatively correlated wit rytmic comlexity. Fritz et al. (2009) resent te most comelling evidence for cross-cultural validity music-emotion maings. Teir subject oulation consisted of twentyone members of te Mafa etnic grou in Nortern Cameroon wo, rior to te study, ad never been exosed to Western music. Subjects were layed musical 29

38 examles meant to convey ainess, sadness, and fear, ranking eac examle by lacing a slider on a continuum between a cartoon ay face and a cartoon grimace. Te Mafa subjects were able to correctly assess te emotional content of eac musical examle, altoug comared to a German control grou, te Mafa resonses were less extreme. After te recordings were digitally altered suc tat formerly consonant armonies became dissonant, te ratings of te Mafa grou became considerably more negative. Wile tis study is emotionally narrow and limited to only two cultures, te results are comelling enoug to lend weigt to te notion tat dynamic emotional signs are understood similarly regardless of acculturation. If musical signs for emotion are understood cross culturally, ten tose signs cannot be ointing toward concets wic are entirely culturally contingent. Tere must be some universally occurring ting to wic emotional musical signs refer. Ekman (1999) suggests emotions are distinctive universal signals for inform[ing] consecifics, witout coice or consideration, about wat is occurring: inside te erson (lans, memories, ysiological canges), wat most likely occurred before to bring about tat exression (antecedents), and wat is most likely to occur next (immediate consequences, regulatory attemts, coing). All kinds of actions are acked into emotions: tose wic lead u to te emotional exerience, tose wic are a art of or coincide wit its occurrence, and tose to wic it is an antecedent. Following tis observation, it seems likely tat cross-cultural emotions are accomanied by redictable atterns of beavior. Examles include figting or yelling wen angry, receding or crying wen sad, or moving slowly and becoming still wen eaceful. We would like to suggest tat emotional signs in music bear an iconic relationsi to tese and similar activities, including bot ysical actions and modes of seaking affected by emotional state. Tere is some inconclusive 30

39 evidence tat emotions and actions may be associated in a way wic would suort tis semiotic relationsi. In addition to te studies of movement and gesture summarized above, Ekman (1999) summarizes studies wic indicate emotions may be reliably accomanied by certain ysiological canges wic could redisose subjects to certain activities. None of te studies summarized offer evidence as to weter tese redisositions are innate or te result of acculturation, so tis is still an oen question. However, if evidence of crosscultural validity of emotional signs in music is sown to be conclusive, tat would strongly suggest tat emotional redisositions to beavioral action are cross-cultural as well. 2.7 Were do we go from ere? Evidence from te literature sows consistent maings between music, motion, and emotion wic aear to be determined by cross-modal arametric isomorisms and inter-arametric analogies. However, tis evidence for crossmodal maings is mostly imlicit: most of te studies surveyed focus on examles of emotion-signifying stimuli in a single modality wic are eiter created rior to te exeriment or by actors. Subjects tyically assess te emotionality of te stimulus, and ten te exerimenters arameterize and analyze tose stimuli wit resect to subjects' judgments. Te rimary roblem wit tis aroac is te segregation of different modalities. Subjects only assess stimuli in a single modality at a time, and researcers tyically analyze eac modality searately, so relationsis between modalities are rarely described in detail. 4 In te case of music, tis roblem is comounded by a willingness to take music-teoretical terminology (esecially major and minor ) as basic asects 4 Eitan and Granot (2003) and (2006) are notable excetions. 31

40 of musical exerience, obscuring te ossibility tat lower-level arameters could be at lay. Finally, te use of uman actors for roducing stimuli imoses severe limits on te number of stimuli generated and te extent to wic tose stimuli reflect te full breadt of exressive ossibility. 3 A beavioral exeriment Our exeriment avoids te metodological issues described above by inverting te standard create stimuli, ten get subject assessments, ten analyze rocess. We develoed a novel exerimental aradigm were subjects are resented wit a comuter rogram wic allows tem to maniulate slider bars corresonding to arameters in a statistical model generating dynamic contours. Te outut of tis model is fed to comuter rograms wic simultaneously create stimuli in two different modalities music and motion in real time, wit similar dynamic contours. Two grous of subjects, one for eac modality, use te model as an autoring tool, creating stimuli wic tey tink best exress a set of emotions. Afterward, te results from te two grous are comared. If te same statistical roerties of dynamic contour are similarly imlicated in emotion recognition in bot music and motion, ten effect of class (emotion) on slider osition sould be tan te effect of modality (music, motion). In addition to avoiding te sortcomings described above by roviding recisely (in fact, rogrammatically) exlicated definitions of terminology, tis aroac also results in te roduction of a generative model for creating numerous statistically and emotionally similar stimuli. 32

41 3.1 A notable metodological recedent In addition to te literature described above, tere is one study wit wic te resent work sares a certain kinsi. Its nearest metodological neigbor is Scerer and Osinsky (1977), wic is te first study of musical emotion to use stimuli generated based on regular samling of a musical arametric sace. Scerer and Osinsky used a Moog syntesizer to create eigt-tone melodies based on te division of musical sace into various arameters, eac of wic was furter divided into levels of intensity. Because te melodies were roduced manually, tey were limited in terms of te resolution of teir divisions, wic in turn limited te number of stimuli roduced (164 in total, wic were narrowed down to a smaller grou for testing). Wile tis is quite a large selection relative to oter contemoraneous studies, te resent work exands tis furter by automating te melody generation rocess and anding control of te arameter settings to te subjects, enabling fine-grained exloration of a very large arametric sace. 3.2 Parameterization and emotion coices Based on a review of te literature outlined above, we selected five emotions on wic to focus our researc. We cose emotions likely to be recognizable in bot music and simle movement. Tese emotions were reresented in our study by five-word clusters, following Hevner (1935). Eac cluster is toed by a single word we decided was te clearest and simlest exression of te emotion-grou. Tese to words are: ay, sad, angry, scared, and eaceful. We ten selected a grou of arameters imlying a model we tougt could reresent simle music and biological motion. Tis model was te basis of te stimulus-generating comuter rogram described in te following section. By 33

42 simle music and biological motion, we mean to indicate our goal was not realism, but mere recognizability; te musical outut of our rogram is not going to sit alongside Mozart in te canon, but sould be recognizable as ay music, sad music, etc. Likewise, te rogram does not generate realistic uman movement, but instead bounces a ball around, te motion of wic sould be recognizable as ay, sad, and so on. 5 A combination of evidence from te literature and our intuition suggested te arameters of temo or interonset interval (measured in beats er minute), jitter (standard deviation from te mean temo), musical consonance/visual sikiness, tendency to make big movements vs. small movements, and tendency to move uward or downward. Eac of tese arameters are isomoric in bot music and motion, wit te excetion of consonance/sikiness. For consonance/sikiness we imlemented a maing from a simle model of musical consonance to te visual sikiness of te moving figure. 3.3 Te model Te rogram for te beavioral exeriment was written in Max/MSP (Puckette, 1991; Zicarelli, 1998), JavaScrit, and Processing (Reas & Fry, 2006). Subjects were resented an interface wit slider-bars corresonding to te five dimensions of our statistical model: temo, jitter, or scale coice (also referred to as dissonance or consonance), ste size, and ste direction. We selected tese arameters based on intuition and exerience, augmented by a review of te literature, identifying eac arameter as eiter crucial for emotional exression or as low-level ground uon wic iger-level ideas migt 5 Tat is to say, if tis model is an exlication of music, motion, or emotional dynamics, it a umble, ragmatic one. Its aim is to balance recognizability wit ease of use, not to be te most accurate model in town. 34

43 deend (dissonance, for examle, deends uon scale coice). Te five sliders controlled arametric values fed to an algoritm wic robabilistically moved te osition of a marker around a discrete number-line in real time. We will refer to te movements of tis marker as a at. Te osition of te marker at eac ste in te generated at was maed to eiter music or animated movement. Te number-line traversal algoritm can be slit into two arts. Te first art, called te metronome, controlled te timing of trigger messages sent to te second art, called te at generator, wic ket track of and controlled movement on te number line. Te temo and jitter arameters were fed to te metronome, and te scale coice (also referred to as consonance), ste size (also referred to as bigsmall), and ste direction (also referred to as udown) arameters were fed to te at generator. Wen te subject ressed te sace bar on te comuter keyboard, te metronome turned on, sent sixteen trigger messages to te at generator (variably timed as described below), and ten turned off. Te beginnings and endings of ats corresond to te ons and offs of te metronome. Temo was measured in beats-er-minute (bm), and constrained to a minimum of 30bm and a maximum of 400bm. Jitter was exressed as a coefficient of te temo wit a range between 0 and Wen jitter was set to 0, te metronome would send out a stream of events at evenly saced intervals as secified by te temo slider. If te jitter slider were above zero, ten secific er-event delay values were calculated nondeterministically as follows. Immediately rior to eac event, a uniformly random value was cosen between 0 and te current value of te jitter slider. Tat value was multilied by te eriod in milliseconds as secified by te temo slider, and ten te next event was delayed by a number of milliseconds equal to te result. Te effect was tat 35

44 as te value of te jitter slider increased, te timing of event onsets became less redictable wile te mean event density remained te same. Te at generator can be conceived of as a black box wit a memory slot wic could store one number and wic resonded to a small set of messages: reset; select next number; and outut next number. Wenever te at generator was sent te reset message, a new starting osition was icked and stored in te memory slot (te exact value of te starting osition was constrained by te value of te scale coice slider as exlained below). Wenever te at generator was sent te select next number message, it icked a new number according to te constraints secified by te slider bars first, te size of te interval was selected, ten te direction (u or down), ten a secific number according to te osition of te scale coice slider. Te outut next number message caused te at generator to outut te next number to te music and motion generators, described in section 3.4. Wen selecting a new number, te at generator first cose a ste size, or te distance between te revious number (stored in te memory slot) and te next. Tis value was calculated nondeterministically based on te osition of te ste size slider. Te ste size slider ad a minimum value of 0 and a maximum value of 1. Wen coosing a ste size, a uniformly random number between 0 and 1 was generated. Tis number was ten used as te x value in te following equation, were a = te value of te ste size slider: Te result r was multilied by 4 and ten rounded u to te nearest integer to give te ste size of te event. As te value of te ste size slider increased, te 36

45 likeliood of a small ste size decreased, and vice versa. If te slider was in te minimum osition, all te stes would be as small as ossible. If it was in te maximum osition, all te stes would be as large as ossible. If it was in te middle osition, tere would be an equal likeliood of all ossible ste sizes. Oter ositions skew te distribution one way or te te oter, were iger values resulted in a larger average ste size. Note tat tese ste size units did not corresond directly to te units of te number line; tey were flexibly maed to te number line as directed by te user's scale selection, described below. After te ste size was cosen, te at generator determined te direction of te next ste: u or down. As wit ste size, te ste direction was calculated nondeterministically based on te osition of te ste direction slider. Te ste direction slider ad a minimum value of 0 and a maximum value of 1. Wen coosing ste direction, a uniformly random number between 0 and 1 was generated. If tat number was less tan or equal to te value of te ste direction slider, ten te next ste would be downward; oterwise te next ste would be uward. Finally, te number was maed on to one of 38 unique scales. As te notion of a scale is drawn from Western music teory, tis decision requires some elaboration. In Western music teory, a collection of itces layed simultaneously or in sequence may be eard as consonant or dissonant. Te ercetion of a given musical note as consonant or dissonant is not a function of its absolute itc value, but of te collection of intervals between all itces comrising te current cord or rase. Te relationsi between interval size and dissonance is non-linear. For examle, an interval of 7 alf stes, or a erfect fift, is considered quite consonant, wereas an interval of 6 alf stes, or a tritone, is considered quite dissonant. Intervallic distance, consonance/ 37

46 dissonance, and equivalency are closely related. If a collection of itc classes X (a itc class set, or PC set) as te same set of intervallic relationsis as anoter PC set Y, tose two PC sets will ave te same degree of consonance and are transositionally identical (and in certain conditions equivalent). Absolute itces also osses tis roerty of transositional equivalency. Wen te frequency of a note is doubled, it is erceived as belonging to te same itc class. For examle, te A key closest to te middle of a iano as a fundamental frequency of 440Hz, wile te A an octave iger as a fundamental frequency of 880Hz; bot are eard as an A. Western music divides te octave into twelve itc classes, called te cromatic scale, from wic all oter scales are derived. Because we wanted to investigate musical dissonance and ossible functional analogs in te modality of motion, our number-line scales were designed to be analogous to musical scales, were a number-line scale is a 5- member subset of te cromatic set [0,1,2,3,4,5,6,7,8,9,10,11]. Tere are 768 subsets of te cromatic set, many of wic are (in te domain of music) transositionally or inversionally equivalent. Our scale list was created by generating te rime forms (Forte, 1973) of tese 768 subsets, and ten removing dulicates, yielding 38 unique scales. 6 Tese scales were ordered by teir aggregate dyadic consonance as defined by Huron (1994). Te coice of a definition of consonance determined exclusively by itcclass relationsis may seem at odds wit our reflection-ercetion feedback model of concetual understanding outlined in section 2.6. However, wile motivated by Western music teory, Huron's (1994) aggregate dyadic consonance is a ercetual measure. It is derived from te results of tree searate studies (Malmberg, 1918; Kameoka and Kuriyagawa, 1969; Hutcinson 6 Tere are 38 rime 5-member PC sets, but only 35 unique interval vectors, so two of te entries on our slider bar were redundant. 38

47 and Knooff, 1979; all as cited by Krumansl, 1990) in wic subjects were surveyed as to te relative dissonance of various combinations of notes layed on a iano. We tink tat tis, a measure based solely on tese subjective judgments, in a context wic closely matces tat of our exeriment, is te best we can do to balance te need for exlication imosed by comuter modeling wit our need to rely on a folk notion of dissonance. Tis aroac is not witout its limits: Huron's metric is only alicable to listeners acculturated to Western music, and does not take into account te effects of melody or itc order, loudness, itc register, or any musical arameters oter tan interval class. Wile tis limits to some extent te generalizability of our results, and te alicability of te exeriment to oter cultural contexts, we believe it is sufficient for te resent work. Te algoritm for generating a secific at across te number line was as follows. Te number line consisted of te integers from 0 to 127 inclusive. Wen te algoritm began, tree variables were stored. First, a starting-oint offset between 0 and 11 was selected uniformly at random, ten an octave bias variable was set to 5, and a scale osition variable was set to 0. Te current scale class was determined by using te scale osition variable as an index to te array of scale elements secified by te osition of te scale slider. For examle, if te current selected scale was [0, 3, 4, 7, 10] and te current scale osition variable was 2, ten te current scale class would be 4 (indices start from 0). Te current osition on te number line was given by multilying te octave bias by 12, adding te starting-oint offset, and ten adding te current scale class value. For examle, if te octave bias was 5, te starting-oint offset was 4, and te scale class value was 7, ten te current osition on te number line would be

48 Wen te select next number message was received, an interval and note direction were selected as described above. If te note direction was uward, ten te new scale osition value was given by te following: (current scale osition + new interval value) % 5 If te note direction was downward, ten te new scale osition value was given by: 5 + (current scale osition - new interval value) Eiter of tese conditions may imly a modular wraing around te set of ossible values (0 to 4). If tis is te case, ten te current octave variable is eiter incremented by 1 in te case of an uward interval, or decremented by 1 in te case of a downward interval. If a ste in te at would move te osition on te number line outside of te allowed range, 12 would be eiter added to or subtracted from te new osition. Tis to some extent distorted te contour of ats wit very large ste sizes wic ad an extreme tendency toward eiter uward or downward movement. 3.4 Te maings Te subjects were divided into two grous. For te first grou, number-line values were maed to musical notes, and for te second grou, number-line values were maed to animated movement. 40

49 3.4.1 Music Our maing from movement across a number-line to Western music was straigtforward, as its most significant modality-secific features were taken care of by te very design of te number-line algoritm. Te division of itces into itc-classes and scales is accounted for by te scale-class and scale selection system used by te algoritm, as is te modulo 12 equivalency of itc-classes. Eac number was maed to a secific itc wic was sounded as te algoritm selects te number. Te number 60 was maed to middle-c, or C4. Movement of a distance of 1 on te number line corresonded to a itc cange of a alfste, wit iger numbers being iger in itc. For examle, 40 mas to E2, 0 mas to A0, and 127 mas to G9. Tis matces te maing described by te MIDI Manufacturers Association (1996). Notes were triggered via MIDI and layed on te grand iano instrument included wit Ale GarageBand. A iano timbre was icked because of te instrument's ubiquity in Western music and relative emotional neutrality. Unlike te violin, guitar, clarinet, etc., te iano aears in almost every genre of Western music, and is routinely used to exress te full sectrum of musical emotions. Te violin or cello, for examle, could for some listeners carry a connotation of sadness. Furter, te iano does not necessarily carry any extra-musical connotations unlike, for examle, a ure sine tone, wic is often used to signify te future or advanced tecnology. Tis is not to suggest te iano rovides a truly neutral timbre, or tat it cannot be used to oint in an extra-musical direction, but simly to say tat no emotional information or extra-musical context may be reliably inferred from its use. 41

50 3.4.2 Motion Maing from movement across a number-line to animated movement was less straigtforward. Our animated caracter was a red ellisoid egg wit cubic eyes. It sat ato a rectangular dark grey floor on a ligt grey background. An ellisoid was cosen because it can be seen as rotating around a center. Te addition of eyes was intended to engage cognitive rocesses related to te ercetion of biological motion. We wanted our subjects to erceive te egg as aving its own subjectivity; tat it could be caable of communicating or exeriencing ainess, sadness, etc. Te movement of our caracter (encefort referred to as te Egg ) was limited to bouncing u and down, rotating forward and backward, and modulating te sikiness of its surface. Tecnical details follow. Te Egg was rendered using OenGL (Rost, 2004) and rogrammed using Processing (Reas & Fry, 2006). Te Egg was drawn as a red 3-dimensional sere comosed of a limited number of triangular faces wic were transformed into an ellisoid by scaling its y-axis by a factor of 1.3. Te Egg was ositioned suc tat it aeared to be resting on a rectangular floor beneat it. Its base aeared to flatten were it made contact wit te floor. Te total visible eigt of te Egg wen it is above te floor was 176 ixels; tis was reduced to 168 ixels wen te Egg was making contact wit te floor. Its eyes were small wite cubes located about 23% downward from te to of te ellisoid. Te Egg and te floor are rotated about te y-axis suc tat it aeared te Egg was looking somewere to te left of te viewer. 42

51 Figure 1. Asects of te Egg 43

52 Every time te current osition on te number line canged, te Egg bounces. A bounce is te translation of te Egg to a osition somewere above its resting osition and back down again. Bounce duration was equal to 93% of te current eriod of te metronome. Te 7% reduction was intended to create a ercetible landing between eac bounce. Bounce eigt was determined by te difference between te current osition on te number line and te revious osition. A difference of 1 resulted in a bounce eigt of 20 ixels. Eac additional addition of 1 to te difference increased te bounce eigt by ixels, e.g. a difference of 5 would result in a bounce eigt of ixels. Te Egg reaced its translational aex wen te bounce was 50% comlete. Te arc of te bounce followed te first alf of a sine curve, i.e. at any oint during te bounce, te current vertical translation of te Egg relative to its original osition was given by te formula: Were is a decimal value between 0 and 1 reresenting te ercentage of te bounce comleted and is te total eigt of te bounce. Te Egg would rotate, leaning forward or backward, deending on te current number line value. Hig values caused te Egg to lean backward, suc tat it aeared to look uward, and low values caused te Egg to lean forward or look down. Wen te current value of te number line was 60, te Egg's angle of rotation was 0 degrees. An increase of 1 on te number line decreased te Egg's angle of rotation by 1 degree; conversely, a decrease of 1 on te number line increased te Egg's angle of rotation by 1 degree. For examle, if te current number-line value were 20, te Egg's angle of rotation would be 40 degrees. If 44

53 te current number-line value were 90, te Egg's angle of rotation would be -30 degrees. Te Egg could also be more or less siky. Te amlitude of te sikes, or erturbations of te Egg's surface, were analogically maed to musical dissonance. Te visual effect was acieved by adding noise to te x, y, and z coordinates of eac vertex in te set of triangles comrising te Egg. Wenever a new osition on te number-line was cosen, te aggregate dyadic consonance (Huron, 1994) of te interval formed by te new osition and te revious osition was calculated. Te maximum aggregate dyadic consonance was 0.8, te minimum was Te results were scaled suc tat wen te consonance value was 0.8, te sikiness value was 0, and wen te consonance value was , te sikiness value was 0.2. Canges in consonance of 0.01 resulted in a cange of to te sikiness value. For eac vertex on te Egg's surface, sikiness offsets for eac of te tree axes were calculated. Eac sikiness offset was a number cosen uniformly at random between -1 and 1, wic was ten multilied by te Egg's original serical radius times te current sikiness value. 3.5 Exerimental Metod Subjects were divided into two grous, te Motion grou and te Music grou. Te same rogram was used for bot grous of subjects, excet tat te Motion grou only saw te motion outut, wereas te Music grou only eard te music. Te rogram was exlained to te subjects as follows: wenever te sace bar was ressed, a musical rase would begin to lay or te ball would begin to bounce. Wile te music was laying or ball was bouncing, a visual indicator would aear on te screen, and additional resses of te sace bar would ave no effect. Moving te slider bars immediately caused te music or te way te 45

54 ball bounced to cange. Subjects were given an oortunity to lay wit te slider bars in an oen-ended way for as long as tey liked. Wen tey were ready, tey were instructed to ress a button on te screen wic dislayed te emotional targets (exlained below) and began te exerimental task. Five emotional targets were dislayed on te screen. Eac target consisted of a grou of five emotional words, a save button, and a load button. Te targets aear in figure 2, a screensot of te user interface. Figure 2. Screensot of te user interface Pressing te save button for a grou stored te slider bar ositions in memory. Pressing te load button for a grou restored te slider bars to te osition saved for tat grou. Subjects were instructed to save slider bar settings corresonding to eac emotional grou. No grou order was mandated, tat is, subjects were free to work on itting eac emotional target in watever order tey cose, and tey were instructed to load and revise settings as canges occurred to tem. Eac saved slider bar setting was meant to make sense wit resect to te oter saved settings, i.e. te subject was told tat teir ay settings sould make 46

55 sense relative to teir sad settings, and so fort. Te difference between recognized and evoked emotion was emasized: subjects were told tat teir task was to make te emotions recognizable to an observer but tat tey sould not worry about trying to make te clis emotionally evocative. Tere was no time limit imosed on te exeriment; wen subjects saved slider settings for all five emotional targets and were satisfied wit te results, tey ressed a button wic finised te exeriment. 3.6 Results Multi-way ANOVA/GLM Unless oterwise noted, Maucly's Test of Sericity was significant for all witin subject effects. To comensate, all results listed below ave Greenouse- Geisser correction alied. Emotion ad te largest effect on slider osition (F (2.97, ) = , < 0.001). Te artial Eta2 was.79, wic means tat Emotion by itself accounted for 79% of te overall (effect+error) variance. Tis main effect of emotion was qualified by an Emotion x Feature interaction indicating tat different emotions required different configurations of dynamic features (F(4.81, ) = , < 0.001; artial Eta2 =.70). Imortantly, wile tere was a significant main effect of Modality (F(1,48) = 4.66, <.05) tis effect was small (artial Eta2 =.09) and did not interact wit Emotion (Emotion x Modality: F(2.97, ) =.97, >.4; artial Eta2 =.02). Te tree way interaction between Feature, Emotion, and Modality was significant, albeit modest (F(4.81, ) = 4.50, < 0.001; artial Eta2 =.09). Te tree-way interaction can be read as a measure of variance er arameter exlained by te combination of emotion and modality. Tat is, a roug measure of te extent to wic te statistical codes for emotion in our 47

56 model differ between music and motion. Tis measure combines tose differences wic are a function of te uman ercetual system wit differences caused by limitations and inaccuracies in our model. Its modest size suggests, unsurrisingly, tat te domains of music and motion are to an extent fundamentally different, but also tat tey are sufficiently similar, and our models sufficiently accurate, for te uroses of our exeriment Analyses by emotion class Te following sections describe te data for eac emotion in detail. Means wit standard deviations are rovided for eac slider bar and task combination, interarametric correlations wit magnitudes > 0.3 are discussed, and linear discriminant analysis (LDA) is used to assess wic arameters best distinguis data oints in eac emotion class from all out-of-class data oints. To describe te results of LDA, we rovide te roortion of te linear combination of redictor variables wic describe te rotation of te discriminant for eac arameter. Tis is a relatively abstract measure; suffice it to say tat ig values indicate te given arameter is imortant for discriminating te current emotion from te oters. Parameters reresented in te model as values between 0 and 1 are scaled to between 0 and 100. Te ossible ranges of eac arameter are as follows: BPM, ; jitter, 0-99; consonance, 0-37; bigsmall, 0-100; udown,

57 Angry bm jitter consonance bigsmall udown Mean all SD all Mean Music SD Music Mean Motion SD Motion Table 2: Angry: means and standard deviations Music and motion Consonance, udown r = -0.39, <0.005, 95% CI -0.6 to BPM, udown r = -0.32, <0.025, 95% CI to Music Consonance, udown r = -0.39, <0.05, 95% CI to 0 BPM, udown r = -0.33, <0.1, 95% CI to 0.07 Jitter, bigsmall r = -0.43, <0.03, 95% CI -0.7 to Motion Consonance, udown r = -0.39, <0.052, 95% CI to 0 Consonance, bigsmall r = -0.45, <0.023, 95% CI to Table 3: Angry: correlations wit magnitude greater tan

58 music and motion music motion bm jitter consonance bigsmall udown Table 4: Angry: LDA results Music and motion settings for angry were very similar. Anger is quick, jittery, dissonant, takes large stes, and tends to move rater steely downward. Altoug BPM values for motion ad a lower standard deviation tan music, a majority of music subjects used te BPM slider to its igest ossible value. Te minimum BPM value for angry music was 136, wereas te minimum for angry music was 269. Angry motion also tended to be more jittery, wit a mean value of 61.13, vs for music. In bot angry music and angry motion, consonance and udown were negatively correlated, meaning tat as subjects steeened te downward trajectory of te at, tey also decreased te consonance. In music alone, BPM and udown as well as jitter and bigsmall were negatively correlated, meaning tat as subjects steeened te downward trajectory of te at, tey also slowed down te temo, and as tey cose to increase te ste size, tey also cose to decrease te jitter. In motion alone, consonance and bigsmall were negatively correlated, meaning tat as subjects decreased te consonance (and so increased te sikiness of te Egg), tey also decreased te ste size. Te jitter-bigsmall correlation in music and te consonance-bigsmall correlation in motion are of similar size (-0.43 and -0.45) and in te same direction; also, witin te dataset as a wole (bot tasks for all emotions), jitter and consonance are negatively 50

59 correlated (a discussion of te wole dataset correlations is below). Tis leads us to yotesize tat motion subjects' sikiness coices and music subjects' consonance coices are similarly motivated. Angry data oints are best discriminated from oter data oints by assessing teir osition on te consonance-udown lane. Consonance is te most imortant arameter (0.577), followed by udown (0.355). LDA results for angry motion alone are similar but exaggerated. For angry music alone te result is different: consonance and udown are still te most imortant dimensions for distinguising angry from te oter emotions, udown is more imortant (0.634) tan consonance (0.265) Hay bm jitter consonance bigsmall udown Mean all SD all Mean Mu SD Mu Mean Mo SD Mo Table 5: Hay: means and standard deviations 51

60 Music and motion BPM, jitter r = 0.51, <0.0001, 95% CI 0.27 to 0.69 BPM, bigsmall r = -0.33, <0.02, 95% CI to Music BPM, bigsmall r = -0.36, <0.08, 95% CI to 0.04 Udown, bigsmall r = 0.4, < 0.043, 95% CI 0.02 to 0.69 Motion BPM, jitter r = 0.62, <0.002, 95% CI 0.29 to 0.81 Table 6: Hay: correlations wit magnitude greater tan 0.3 music and motion music motion bm jitter consonance bigsmall udown Table 7: Hay: LDA results Hay is fast, sligtly jittery, consonant, as medium ste size, and tends moderately uward. Tere are a number of significant differences between ay music and ay motion. Hay music tends to be faster. Te BPM value for ay motion as a large standard deviation and a relatively low minimum value of 98; it's ossible tat motion may aear ay so long as it is above a certain tresold. Hay music tends to be more jittery tan motion. 8 of 15 ay music subjects cose jitter values of 26 or below, wile te rest cose values between 48 and 95. It may be tat te ig-jitter subjects were trying to create more elaborate rytms. 52

61 In ay music and motion considered togeter, BPM and bigsmall are negatively correlated, meaning tat as seed increases, ste size decreases. Tis suggests tat subjects want to acieve an increase in seed wic does not also dramatically increase te at's angle of uward movement. Peras because tis angle of uward movement is robably less ercetible in te motion domain tan te music domain, tis correlation is stronger in music (r = -0.36, vs. r = -0.13). Tis yotesis is strengtened by te correlation in music of udown and bigsmall (r = 0.4), demonstrating tat as ste size increases, te ratio of uward to downward movements decreases, moderating te angle of uward movement in te same way as te BPM-bigsmall correlation. Across bot tasks, BPM and jitter are relatively strongly correlated, suggesting tat as temo increases, more jitter is necessary to acieve te same effect. Wen only considering ay music and motion, tis correlation is muc stronger in motion (r = 0.62) tan in music (r = 0.22). Tis may be because of differences in accuracy in visual vs. auditory rytm rocessing (Saenz and Koc, 2008). Tis correlation isn't unique to ainess: it olds across all emotions for bot tasks (r = 0.45, <5.33e-13, 95% CI 0.35 to 0.54), for just te music task (r = 0.39, < 5.69e-06, 95% CI 0.23 to 0.53), and for just te motion task (r = 0.52, <6.801e-10, 95% CI 0.37 to 0.63). Consonance is by far te most imortant dimension for distinguising ay music and motion data oints from te oter emotions. For music and motion togeter, te second most imortant dimension is udown (0.06), and udown is still more imortant for music on its own (0.164). For motion alone, owever, bigsmall is more imortant (0.071) tan udown (0.011). 53

62 Peaceful bm jitter consonance bigsmall udown Mean O SD O Mean Mu SD Mu Mean Mo SD Mo Table 8: Peaceful: means and standard deviations Music and motion Bigsmall, consonance r = -0.31, <0.026, 95% CI to Bigsmall, jitter r = 0.33, <0.018, 95% CI 0.06 to 0.56 Consonance, jitter r = -0.49, < , 95% CI to Music Consonance, jitter r = -0.32, <0.13, 95% CI to 0.09 Motion Bigsmall, consonance r = -0.5, <0.012, 95% CI to Bigsmall, jitter r = 0.56, <0.0036, 95% CI 0.21 to 0.78 Consonance, jitter r = -0.68, < , 95% to Table 9: Peaceful: correlations wit magnitude greater tan

63 music and motion music motion bm jitter consonance bigsmall udown Table 10: Peaceful: LDA results Peacefulness as a slow or slow-medium temo, very low jitter, is quite consonant, takes small stes, and tends uward. Peaceful music tends to be faster tan eaceful motion, wit a mean BPM of 73.6, witin te average normal uman eart rate range of (Mancia et al., 1983). Across bot tasks, consonance and jitter are negatively correlated, bigsmall and jitter are ositively correlated, and bigsmall and consonance are negatively correlated, suggesting tat dissonance, jitter and large ste size are similar insofar as tey work to disrut te eace. Te correlation between bigsmall and jitter in eaceful music alone is very weak (r = 0.04); rater tan suggesting a fundamental difference between te tasks, tis may because te standard deviation of jitter values in eaceful music is so dramatically small, i.e. jitter so effectively disruts eacefulness tat in te music task subjects escewed it almost comletely. Tis effect may be more extreme in music because of increased rytmic acuity in audition vs. vision (Saenz and Koc, 2008). Peaceful music and motion togeter are best distinguised from te oter emotions by osition on te consonance-udown lane, altoug udown (0.523) is more imortant tan consonance (0.378). Peaceful motion is similar. For eaceful music on its own, consonance is te most imortant dimension (0.573), followed by bigsmall (0.178) and udown (0.132); osition on te 55

64 consonance-udown lane alone is not enoug to distinguis eaceful music from oter kinds of music, tree dimensions are necessary Sad bm jitter consonance bigsmall udown Mean O SD O Mean Mu SD Mu Mean Mo SD Mo Table 11: Sad: means and standard deviations Music and motion Udown, bigsmall r = -0.35, <0.013, 95% CI to Udown, consonance r = 0.38, <0.007, 95% CI 0.11 to 0.59 Jitter, consonance r = -0.42, <0.0022, 95% CI to Music Jitter, udown r = -0.36, <0.079, 95% CI to 0.04 Jitter, consonance r = -0.47, <0.018, 95% CI to Motion Udown, bigsmall r = -0.63, < , 95% CI to Udown, consonance r = 0.42, <0.036, 95% CI to 0.7 Jitter, consonance r = -0.44, <0.029, 95% CI to BPM, consonance r = 0.31, <0.13, 95% CI to 0.63 Table 12: Sad: correlations wit magnitude greater tan

65 music and motion music motion bm jitter consonance bigsmall udown Table 13: Sad: LDA results Sadness is slow, as low jitter, is moderately dissonant, takes small stes, and moves decisively downward. Sad motion as a smaller ste size (mean: 1.26; sd: 2.91) tan sad music (mean: 48.4; sd: 31.56), and sad music tends downward at a muc slower rate tan sad motion. Sad motion tends to be quite a bit more consonant tan sad music, wit a majority of subjects lacing te slider at te most consonant osition. Tis indicates our analogical maing of consonance to sikiness is inaroriate for sadness in te context of our model. Tis makes intuitive sense; sikes seem angry or active, and sadness is sedate and slow moving. We suggest an imrovement of te analogical maing would rovide a more natural result, e.g. instead of simly maing note-to-note consonance to sike lengt, it could also ma to sike sarness/dullness, so intermediate levels of dissonance would roduce angles and bums but not sikes er se. It may also be ossible tat our maing is correct, in wic case tis result could demonstrate a fundamental difference between te two domains. Across bot tasks, and in eac task individually, jitter and consonance are negatively correlated, suggesting tat increasing jitter and decreasing consonance serve a similar function for sadness. Across bot tasks, udown and consonance are correlated, meaning tat te steeer te downward trajectory te more consonance is alied. Udown and bigsmall are negatively correlated, 57

66 meaning tat as te downward trajectory becomes steeer, te ste size becomes smaller. Similarly, in te music task, jitter and udown are negatively correlated. In eac of tese tree correlations, te setting in one arameter seems to moderate te effects of te oter. In te motion task, BPM and consonance are weakly correlated. Sad music and motion togeter are best distinguised from te oter emotions based on consonance-udown lane osition, wit udown as te most imortant dimension (0.708). Sad music alone is also based distinguised on te consonance-udown lane, but consonance is te most imortant dimension (0.545), followed by udown (0.323), and ten BPM (0.119). Sad motion alone is best distinguised from te oter emotion based on udown (0.829) and bigsmall (0.108) Scared bm jitter consonance bigsmall udown Mean O SD O Mean Mu SD Mu Mean Mo SD Mo Table 14: Scared: means and standard deviations 58

67 Music and motion Music None None Motion Udown, jitter r = 0.34, <0.1, 95% CI to 0.65 Table 15: Scared: correlations wit magnitude greater tan 0.3 music and motion music motion bm jitter consonance bigsmall udown Table 16: Scared: LDA results Scared is fast, quite jittery, quite dissonant, and tends neiter uward nor downward. Scared music tends to ave a moderately large ste size, wereas scared motion as a medium-small ste size. Tis difference seems like a fundamental difference between te two modalities; wile bot modalities seem to deend on moment-to-moment unredictability (equal robability of uward and downward motion), scared movement seems tentative, as if walking troug a aunted ouse, wereas scared music is more active, as if being cased by some active treat. Te data for scared are almost entirely uncorrelated, wit te excetion of udown and jitter in scared motion. In every case, consonance is te most imortant dimension for distinguising sadness from te oter emotions. In music and motion togeter, udown (0.06) and jitter (0.04) are also imortant; in music alone, jitter (0.088), 59

68 bigsmall (0.074) and udown (0.034) are imortant, and in motion alone bigsmall (0.11) and udown (0.116) are imortant Wole dataset analyses Music and motion BPM, jitter r = 0.45, <5.33e-13, 95% CI 0.35 to 0.54 BPM, bigsmall r = 0.35, <1.363e-08, 95% CI 0.23 to 0.45 BPM, consonance r = -0.32, <2.04e-07, 95% CI to -0.2 Consonance, jitter r = -0.44, <1.79e-13, 95% CI to Consonance, bigsmall r = -0.33, <1.16e-07, 95% CI to Music BPM, jitter r = 0.39, < 5.69e-06, 95% CI 0.23 to 0.53 Motion BPM, jitter r = 0.52, <6.801e-10, 95% CI 0.37 to 0.63 BPM, bigsmall r = 0.5, <1.67e-09, 95% CI 0.36 to 0.63 BPM, consonance r = -0.48, <1.21e-08, 95% CI -0.6 to Consonance, jitter r = -0.58, <8.074e-13, 95% CI to Consonance, bigsmall r = -0.35, <4.89e-05, 95% CI -0.5 to Table 15: Wole dataset: correlations wit magnitude greater tan 0.3 Across te wole dataset, tere are five moderately correlated arameter airs. As subjects increase te temo, tey tend to decrease consonance, increase jitter, and increase ste size. As subjects increase consonance, tey tend to decrease jitter and decrease ste size. Wen te data are limited to te music grou alone, tere is only one significant correlation wit a magnitude greater tan 0.3: as temo is increased, jitter is increased. Te motion grou is correlated like te 60

69 combined dataset, excet te sizes of te correlations are sligtly larger. Tere U-Down are more significant inter-arametric correlations witin motion tan music. a ss sf a f a af a s a s aa f f a aa s a a f f af a f f a af s a af s a a s aa f f a f 40 a a ss sf a s s f f f f f f f s f f a s s aa s s a f f f 5 f s sf ss s ssf s s s s f a a aa sf f f f s f f f f sf s f af a f s f a a s s a s f f s f s f s Consonance Figure 3: All oints on te consonance-udown lane. a = angry, = ay, = eaceful, s = sad, f = scared

70 Consonance U-Down Wen te dataset is considered as a wole, eac emotion is best discriminated from te oters on te consonance-udown lane. Tis is not to suggest te oter dimensions are unimortant, or tat examining osition on te consonance-udown lane is sufficient to accurately determine te emotion class of a data oint. As te figure 4 sows, desite te results of LDA, discriminating 62 Figure 4: Hay and eaceful data oints on te consonance-udown lane. = ay, = eaceful

71 between ay and eaceful is imossible in terms of consonance and udown. LDA assumes te data can be modeled by gaussian distributions. Tis is more-orless true for angry, sad, and scared, but not true for ay and eaceful, witin wic te ositions of te consonance slider bar are clumed u around te maximum. To determine te arameters wic best distinguis ay and eaceful, we looked at te witin-class covariance of eac arameter wit itself, for bot emotions. Hay Peaceful BPM Jitter Consonance Bigsmall Udown Table 18: Witin class covariance for ay and eaceful Relatively large witin-class covariance indicates te data oints clum togeter in tat dimension, suggesting its imortance for between-class discrimination. For bot ay and eaceful, witin class covariance is igest for BPM and bigsmall. And, indeed, lotting ay and eaceful togeter on te bigsmall- BPM lane (figure 5) makes it ossible to cleanly discriminate between te two emotions. 63

72 Big-Small BPM As seen in table 19, consonance was found to be te arameter most imortant for discriminating between music and emotion data oints. Tis indicates tat, in relative terms, music subjects used consonance differently tan motion subjects used sikiness. Examining te data in absolute terms, consonance values 64 Figure 5: Hay and eaceful data oints on te bigsmall-bpm lane = ay, = eaceful

73 across modalities are quite similar. A ig LDA imortance value may be one way of distinguising between analogical and isomoric maings. BPM Jitter Consonance Bigsmall Udown Table 19: Music versus motion LDA Similarity analysis/ierarcical clustering Te distance matrix in figure 6 was created by taking te Euclidean distance from every data oint to every oter data oint. Te data were not regularized. Broad structural features of te data are readily aarent: angry, ay, and scared are similar to one anoter, and eaceful and sad are similar to one anoter. Because BPM as te largest range of all of te arameters, but is not always te most imortant for distinguising between emotions, te distances between emotions at different BPM levels seem exaggerated. Te regularized distance matrix in figure 7 rovides more detail. 65

74 Figure 6: Raw distance matrix Were, in te raw distance matrix (figure 6), ay, angry, and scared formed one almost indistinguisable block, and eaceful and sad formed anoter block, in te normalized matrix eac emotion is distinct. We can see tat ay and eaceful are more similar tan ay and sad. Peaceful reveals itself as te most self-similar emotion, wile scared is te least self-similar. 66

75 Figure 7: Regularized distance matrix A dendrogram (in miniature in figure 8, and available in full online at tt://beausievers.com/tesis2010/dendrogram/) was generated from te regularized distance matrix, using te average distance between oints as te affinity function. Te clustering was erformed by R's (R Develoment Core Team, 2008) clust function, wic, at eac merge, laces te tigter subtree at 67