Music, Timbre and Time

Music, Timbre and Time Júlio dos Reis UNICAMP - julio.dreis@gmail.com José Fornari UNICAMP tutifornari@gmail.com Abstract: The influence of time in music is undeniable. As for our cognition, time influences how we sense acoustical and musical events. Here we divide the process of listening in three steps: Perception, Cognition and Affect. Each of them occur through different brain processes and are sensed by human audition in distinct time scales. As subjective processes, they present large perceptual differences from individual to individual. Nevertheless, it is still possible to study an average sense of time and duration for the perception, cognition and affect related to musical processes. This article presents an overview followed by a brief discussion of an ongoing research concerning the philosophical and pragmatic bases for the influence of time in the processes of perception, cognition and affect of music. In this early stage of research, we present a theoretical discussion about temporal processes in music. The aim here is to create the epistemological basis for a further empirical study using computational models to investigate the different time bases for the human perception of time and duration in music. Keywords: time, duration, perception, cognition, affect, timbre 1. Introduction Sound is one of the most important channels of human communication. As such, it has always been the subject of great fascination and phenomenological research on the processes comprising it, from its formation to how it is perceived by the human mind. In particular, musical sound events in nature are often obliterated by this mist of mystery. Several researchers have been investigating physical, physiological and psychological aspects of sounds that compound the art of music. However, despite the advances on this field of investigations, there are still meaningful gaps of scientific understanding, yet to be solved. 1.1. Relation between time and music: In this project, we consider that time and music are conceptually interveners and interwoven. However, one must realize that the relationship between music and time is not purely physical. Acoustical studies always relate to the sound propagation time in windows of seconds (s) or milliseconds (ms), but for musicians such measures are unclear and sometimes are not well classified by the traditional musical notation. For example, in commercial music handling time macrostructural is not explored. Generally the tempo of a popular song remains the same throughout its execution. In this study, however, the settings and extended concepts of musical time, and relate to better analyse time and music. The physics of acoustics defines time in terms of its duration, as a continuous function of time, with variant and measurable magnitude, although it is not possible to address time, by itself, but rather, the modification of other variables and processes. Music measures of time are often not described in the metric system, in seconds, but in subdivisions of a pulse (beats per seconds, or bpm), previously established by the 1283

composer, conductor or performer. As the time for historians, music time is seen as the element that determines the modification of a hierarchically organized network of sound events (eg motives, phrases, passages, etc.). We can conclude that, for music, it is reasonable to think of time as an organizing element of sound events, causing them to be both internally consistent, within a small time scale, and externally segregated, for longer time interval [TENNEY 1980]. In this study it was necessary to utilize these two time settings, because some processes are strongly linked to music listening, or even inevitably linked to the musical context and the musical work. It is also given here a brief citation to the concepts of duration and time established by Henri Bergson, in his paper "Time and Free-will"[BERGSON, 1910], where he defines Time as a sequence of simultaneity, divisible and therefore measurable, and Duration as the single pass and pure time, being indivisible and immeasurable. We believe that his concepts of time and duration are of great importance for further studies of musical affection. The first problem that we confronted is the definition of the boundaries between disciplines that deal with information and musical sound. For instance, how to establish a frontier between the physics of acoustic wave and the physiology of sound sensation? How to separate sound perception studied by psychoacoustics from the human psychology of music cognition? How to separate the evoked emotions from the appraised emotion, while listening to or making music? These are questions that probably we are still far from solving, and perhaps, as in modern conceptions of music, there is no single or definitive way to formalize them. 1.2. Brief definitions of sound hearing process: Under an anthropocentric perspective, we focus on the three mental processes that occur during music listening. They are: Perception, Cognition and Affect. Aspects of perceptual or psychoacoustic, sound phenomena are related to micro-time, context-free, that describe the way information is captured sound like acoustic stimulus (external) for the binaural system (formed by the pair of ears). Music Cognition deals with contextual aspects of sound information, describing with internal, or mental, processes, knowledge and recognition of a given event or musical timbre. These are formed by sound events that occur under a broad duration of time, enough to constitute a memory of its occurrence, thus in music, corresponding to its musical meaning. It is during this process that the identification of an instrument timbre occurs, for example, through perception we notice the sonority of a violin, but it's through cognition that we identify the contextual characteristics of this sonority in contrast to other known, as, in this case, for instance, if this violin is (or not) a Stradivarius. The third process that involves the sound heard (and its musical context) is 1284

Musical Affect. Some studies include this process in the field of music cognition, however, cognition studies the emotion appraised, while musical affect studies the emotion evoked. Affection has significant importance for musical composition and performance. Several books and orchestration works define Timbre, specially the ones of musical instruments, based on these possible emotional state that it can represent, such as: euphoria or solemnity. In fact, musical affect is still a recent and yet to be explored area of human knowledge. Recently, neuroscience has made significant progress in this field, albeit modest, in order to broad the understanding of the complex intervenience of musical events on evoked emotions. 2. Epistemological Bases of Time and Music To better understand musical perception, cognition and affect, we first need to consider that they occur simultaneously, and the separation in different topics is merely to facilitate the study. Next, it is thoroughly defined the concepts of each process and its corresponding timescale, still based on the concepts of time, as mentioned above. 2.1. Perception, or Psychoacoustic process: The sound perception occurs in minute time intervals, in a range known as the psychoacoustic hearing persistence; around 100ms (100 milliseconds) [ROOT 1928]. In this scale of time duration, there are no mental models able to register (memorize) the sound event, thereby making them free of any musical context, despite their great importance for the hearing process. It is in this small order of time interval that occur the identification of the timbre of a sound. When an instrument generates a tonal sound (a sound with clear pitch), the vibrations of the elastic medium (usually air) that make up this signal, do not start with the maximum amplitude, but require some period of time to rise. The mass inertia of the power oscillator which generates the sound signal in a traditional musical instrument (e.g. a string, a membrane, a reed, etc.) requires a period of time necessary for the establishment of the sound event which corresponds to the non-periodic transitions, from the stationary lack of sound to its oscillatory steady-state. This initial transition, from silence to peak amplitude, is named Attack Transient [MENEZES 2004]. Several researches prooved that removing this transient, the auditory identification of timbre is impaired and a sound may even seems to resemble another one, or be considered artificial. The technique of removing the attack of a sound is often used in electroacoustic music and brings the sensation of an acoustic sound to seem like being purely electronic, i.e., generated by mathematical or computational models. Although we should require more experiments to proove it, evidences suggest that the window (or, the duration of) time for the attack transient is under the auditory persistence 1285

(100ms). An interesting aspect to note is the relationship between tone (pitch) and rhythm (beat) and how they are the extremes of the same continuous dimension of perception. The continuity from one to the other is described by the duration of the musical event. It is well known that if a rhythmic cell is accelerates after a given speed (or frequency), it is no longer perceived as a rhythmic structure, but as a continuous sound event with a clear pitch, which can correspond to a musical note. In fact, it is very difficult to get a drummer to play at such speed, but it is possible to use technological resources to validate this perceptual phenomenon. One can speed up the execution of an audio sample in looping such as a short sample of a speak phrase, a rhythmic cell, or a musical excerpt until it is no longer recognized as before, but arising into a continuous musical tone. It is important to observe that this effect will happen even if the looped sample has no clear tonal sounds or steady partials. It is proved that this effect happens when a sound sample is repeated at a rate faster than twenty cycles per second (20 Hz), when this sound homogeneously turns into a constant pitch, which can thus be represented by a musical note. Next figure depicts its frontier. F ig. 1: Tempor al sound percept ion's frontier. In about 20Hz, or a period of 50ms, the perception of the interval between sound events goes from rhythmic (time domain) to tonal (frequency domain). 2.2. Cognitive process: In Cognitive Sciences there are already a great influence of the individual attention, which is related to short-term memory. We can say that it is the moment of transition between an unaware process (Perception) of hearing, to semi- or fully aware (Cognition) process of listening, where we also gain some control to focus our attention, at some extent, according to our will. It is also in the Cognition process that we complement the identification of the timbre of the sounds we hear, which involves the physical events observed during the perceptual process with the individual knowledge acquired in previous life experiences. This is due to the identification of sound aspects made by the musical mind, in contrast to listener's previous sonic experiences, such as memories of the same sound aspects previously listened. Cognition works within a time frame greater than the one of purely perceptual or psychoacoustic processes. According to the studies attributed to the psychologist William James, when listening to music, we have embedded the concept of Specious Present; the reference of a cognitive time scale where most listeners have the sensation of hearing sound events that are not simultaneous, but within a cognitive time 1286

window where they consider to be the musical "now". We argue that the musical "now" is related to short-term memory, which can vary from individual to individual, from situation to situation, as well as according to the complexity of musical or sonic information [POIDEVIN, 2000]. Some experiments have shown that the sensation of musical "now" occurs in durations in the order of about one to three seconds [LEMAN, 2000]. This also enables the cognitive ability to focus the attention to a particular sound event, while ignoring other sonic threads around us, at least partially. Thus, we are able, for example, to have a conversation with another individual in an environment with other conversations occurring simultaneously, as in a cocktail party. This is due to the partial conscious control we have over the process of cognition, or thought, in contrast to the mechanismsof perception that occurs involuntarily. For example, in the same cocktail party example, if there is an unexpected explosion, we involuntarily pay attention to this sonic event. In general, reactions to the processes of sound perception are involuntary, while the reaction to cognitive processes are mostly voluntary. 2.4. Musical Affection and affective process: Musical Affection is, among all three mental processes discussed here, the one most correlated to music context, and thus has such a range of natural (genetic) and nurtured (learned) variables that makes any sort of statement about it speculative and error-prone. The subjective nature of music, as well as listener's age, gender, sociocultural background and training; are, perhaps, aspects that most influence the Affection. For being too large, this field might yield to differences and unpredictable results. Anyway, we might infer that Musical Affection handles the emotion evoked by music through mechanisms related to its long-term memory. Affective aspects are the ones described in terms of emotional dimensions, such as the Circumplex Model, with brings the dimensions of Arousal and Valence, as constituents of a broad range of emotions appraised or evoked by music listening [RUSSEL, 1980]. Emotions in music are evoked by a prosody of Expectations and listeners' mental capabilities of recognize and establish them, forming a discourse of anticipations [HURON, 2006]. This establishes the boundaries between Cognitive and Affective aspects that describe emotions, as cognition deals with emotion appraisal, while affection deals with emotion arousal. Of course, there are plenty of possible discrepancy of emotions evoked by the same musical passage in different individuals. This is due, among others, to the differences between previously acquired (learned) information, distinct sociocultural backgrounds, which are normally all rounded to the subjective label of personal taste. Given these facts, it is expected that there might be a significant variation in the size of these time windows, especially those ones related to musical affection, while for cognition, as it is usually related to the identification and 1287

understanding, short-term memory tends to suffice to describe it in, which is lastly constituted of aspects occurring within the time windows of musical "now" (from 1 to 3 seconds). Fig. 2: Russel's Circumplex Model. The time scale of the emotion evoked by music is particularly explicit in the expectation theory of David Huron. [HURON 2006] developed this theory from the studies of Leonard Meyer [MEYER 1956]. This research deals with the idea that the emotional content of music arises from mental dynamics constructed by the listener predicting the musical events. According to Huron, while listening to music, we create expectations about upcoming musical events and passages, embedded in the musical discourse. In simple terms, the confirmation of a correct prediction causes a feeling of satisfaction, while attesting that a prediction was wrong, cause discomfort feeling. According to this theory, the affect is generated by the musical prosody of correct and incorrect predictions that the listener automatically makes with respect to musical events of any order, such as: melodic, harmonic, rhythmic, orchestration, processing and sound effects. Different expectations can even occur simultaneously in the same, different or intertwined orders. Some researches suggest that emotional arousal is related to long-term memory, which is the process that allows us to compare larger passages of musical information (above specious time) and, for instance, to discriminate between musical genres, styles and any other specific information about a particular performer or composer. As musical affect is related to the process of listening to a musical discourse, we can relate it with Henri Bergson studies about Time. His definitions seem quite appropriate for us to understand how emotions evoked by music are generated. We start from Bergson's concept that Time is a succession of simultaneities, which, our consciousness gathers and organizes in a virtual space, so that it can be measured. In this sense, Time differs from Duration that is immeasurable and indivisible. Based on that, wee can say that the process of Affection is based on the one of a multiplicity of states of consciousness given by Time. In 1288

short, during music listening, there is the possibility of emotion arousal if the listener's conscience is able to capture moments of that music, and arrange them as to make sense of them altogether. Following Bergson reasoning, our consciousness is placing musical simultaneity in the virtual space of our mind, and thus, music that previously permeated consciousness, starts to interact with other sensations and memories we have, for all states of consciousness dwell the same homogeneous medium of time "transposed" into a form of virtual space. From this concept comes the understanding of the interaction between emotions evoked by long-term memory, which involves also listener's sociocultural "baggage". As we feel this way where these emotional interactions occur, as a homogeneous medium, we can also reckon some sort of magnitude for its evoked emotion. 3. Conclusions Given the facts here theoretically investigated, we can foresee some of the challenges of this study, in the further experimental investigation, so we might reach here some valuable contribution for future investigations. We first checked the interdisciplinarity in which this study is placed, which brought out the need of intersection of areas of knowledge that usually do not interact, such as Music and Physics, despite their interconnections in the process of music generation (e.g. luthiers, sound design, sound synthesis, etc.). Another problem that arose was the difficulty to provide a clear definition of what is the time. That still remains, to a certain extent, inconspicuous, to just in music, but for all areas of knowledge that dare to study it. Therefore we had to use more than one definition of time along this study. Finally, the conclusion that the processes here studied do not occur separately but simultaneously, can lead to a wide variations in experimental results may lead to the difficulty of seeking and predicting patterns of occurrence. It is more effective to separate the processes of Perception and Affect, respectively as: unaware and aware, passing through the intermediate process of Cognition, which seems to be semi-aware. We have no control over the perception process, because our auditory system picks up all sounds in the environment that are within our audible spectrum (20Hz ~ 20000Hz) and sends this information to the auditory cortex, a specialized area of our brain that identify sonic information, sort it and allows us to focus attention on a specific sonic event. During the Affection process, we are able to organize this musical information, as we are dealing with the conscious activity of music listening, where we will be subject to the evocation of a feeling or mood. The study of Affection is important not only to research the process of evoking specific emotions on a listener, but, it's in this process that we can evaluate a musical performance, classify a song as how well structured is it, as well as finding the musical style of a song. Clearly, it all depends on relationships made between the information 1289

received and the knowledge previously acquired during a lifetime of music listening, meaning the sociocultural background of each individual, alone or as part of a community, so we believe that the notion of time given by Bergson is meaningful, where he states that we arrange moments, or episodes, within a virtual space of time that are retrieve by our conscience. We conclude by saying that the time scales related to the process of music listening can be more personal than we might have expected, and can even undergo variations, for each specific case. Even though physically we can set limits for each process, psychologically, these limits are not always respected by consciousness, which, in a large share of cases, this can even be consciously manipulated, in order to enhance the emotional experience of musical appreciation, since that music, as an art, is in fact unpredictable and subjective, given by a prosody of anticipations with deep perceptual, cognitive and emotional effects on individuals. And, at the end, we quote Wilhelm Leibniz, in what concerns physics, mathematics and music in a deep synthesis, when he mentions that: Music is the pleasure the human mind experiences from counting without being aware that it is counting. References: HURON, D. Sweet anticipation: music and the psychology of expectation.cambridge, MA: MIT Press, 2007. MENEZES FILHO, F. The Musical Acoustics in Words and Sounds.Editorial Atelie. 308 pp. 2004. TENNEY, J., Polansky, L. Temporal Gestalt Perception in Music. S T. Journal of Music Theory, vol. 24, no. 2, pp. 205-241. Autumn, 1980. ROOT, A. R. Auditory persistence, Summation, and fusion in impulse-successive periods.psychological Review, vol. 35 (6), pp. 507-514. November 1928. RUSSELL, J. A. The circumplex model of Affect. Journal of Personality and Social Psychology, vol. 39. pp. 345-356, 1980. BERGSON, H. Time and Free Will: An Essay on the Immediate Data of Consciousness, translated by F. L. Pogson, M. A. London: George Allen and Unwin. 1910. MEYER, L. B. Emotion and meaning in music. Chicago University Press, 1956. POIDEVIN, R. L. The Perception of Time. In: ZALTA, Edward (org.), The Stanford Online Encyclopedia of Philosophy, 2000. http://www.plato.stanford.edu (visited in 2012). JAMES, W. The Principles of Psychology. New York: Cosimo, 2007. LEMAN, M. et al. Correlation of Gestural Musical Audio Cues and Perceived Expressive Qualities. In: Gesture-Based Communication in Human-Computer Interaction. 5th International Gesture Workshop, pp. 40-54. Berlim / Heidelberg: Springer. LNCS Series 1290

Lecture notes in artificial intelligence, v. 2915. 2004. 1291