Timbral description of musical instruments

Transcription

1 Alma Mater Studiorum University of Bologna, August Timbral description of musical instruments Alastair C. Disley Audio Lab, Dept. of Electronics, University of York, UK David M. Howard Audio Lab, Dept. of Electronics, University of York, UK Andy D. Hunt Audio Lab, Dept. of Electronics, University of York, UK ABSTRACT Musicians intuitively describe timbre using adjectives such as bright or clear. This research aims to establish a set of uncorrelated and consistently used timbral adjectives that could be used to control a future synthesis system. Fifteen timbral adjectives were selected from previous studies and refined in a pilot experiment to: bright, clear, warm, thin, harsh, dull, nasal, metallic, wooden, rich, gentle, ringing, pure, percussive and evolving. These were used as rating scales (e.g. bright to not bright) in a listening experiment. Fifty-nine musicians used these adjective scales to describe twelve instrumental samples. The use listeners made of these scales is analysed by various techniques, resulting in the rejection of evolving, metallic, pure, rich, and wooden. Opposition was clearly demonstrated between some words, notably bright with dull, and harsh with gentle. A sub-set of adjectives is proposed as being useful for future synthesis control. BACKGROUND Musicians intuitively describe timbre using high-level timbral descriptors such as mellow, rich, dull, bright, and harsh to indicate the perceived quality or tonal nature of a sound (Howard and Angus, 2001). The controls of most synthesis methods rarely have an intuitive relationship to In: M. Baroni, A. R. Addessi, R. Caterina, M. Costa (2006) Proceedings of the 9th International Conference on Music Perception & Cognition (ICMPC9), Bologna/Italy, August The Society for Music Perception & Cognition (SMPC) and European Society for the Cognitive Sciences of Music (ESCOM). Copyright of the content of an individual paper is held by the primary (first-named) author of that paper. All rights reserved. No paper from this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval systems, without permission in writing from the paper's primary author. No other part of this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval system, without permission in writing from SMPC and ESCOM. the timbre produced, often using low-level control parameters such as fundamental frequency, basic waveform shape, filter cut-off and resonance. If musicians were able to control a synthesiser using timbral adjectives, they could create sounds in a more intuitive manner. Previous Research Timbre s inherent multi-dimensionality is well-known and explored in Grey (1975) and Creasey (1998), and the application of adjectives to its classification is explored by Berger (1965), von Bismarck (1974), and Kendall and Carterette (1993 (1) and (2)). The use of timbral adjectives by musicians provides clues to the reductive mental system of classification and description they are using. Several studies have examined the relationship between high-level descriptors and timbre for specific instruments. Nykänen and Johansson (2003) list ten common timbral descriptors used by Swedish saxophone players. Disley and Howard (2003 and 2004) used similar methods to gather English words describing pipe organ ensembles, and refined these in subsequent listening tests to an uncorrelated and consistently understood subset of descriptors for use in that context: thin, flutey, warm, bright and clear. Other studies have gathered timbral descriptors without musical stimuli. Moravec and Stepánek (2003) collected words from Czech musicians and developed a subset based on frequency of occurrence. Many studies have looked for generally applicable auditory cues for individual timbral adjectives in isolation, and these are summarised on pages 76 to 81 of Creasey (1998). The problem with such studies is that their solutions tend to gravitate toward the same few readily measurable auditory phenomena such as the spectral centroid and relative harmonic strengths. This results in multiple theories difficult to apply simultaneously. Many of these words do not have a single obvious correlation with spectral features, with some words simultaneously describing both timbral quality and perceived loudness (Sandell, 1991, p25). The attempt to define such gen- ISBN ICMPC 61

2 eral relationships is perhaps doomed to failure in all but a small number of cases, as much usage of these words is inherently subjective and dependent on the learning of relationships between timbres and adjectives, an area that has been largely ignored thus far. Von Bismarck (1974) summarises timbral scales from many sources and creates a subset of four (dull-sharp, compact-scattered, full-empty and colourless-colourful) but Kendall and Carterette (1993 (1)) question both the relevance of these scales to real instruments and the wisdom of assuming the opposition of words, going on to demonstrate more success with scales of x to not x. Much of this work also demonstrates problems in applying results outside of its linguistic origin. Disley and Howard (2004) suggest that even different dialects within a common language (US and UK English) may have different understandings of descriptors, and that this difference is not consistent for all timbral descriptors. Warm and clear in particular appeared to have different meanings to US and UK English speakers AIMS This research aims to establish a set of uncorrelated and consistently used adjectives that can be used to control a future synthesis system. The first part of this project must be to survey musicians on the language and terms they use to describe sound. Some previous studies have used musical examples in this process and others have not. The former has the advantage of gathering words which are commonly used in real life situations, whereas the latter could gather many words not used in actuality. The use of samples to trigger listeners use of words requires careful choice of samples to avoid biasing the result. As the end goal is to provide a tool for musicians, trained musicians must be used throughout the experimental procedure, which will help to allay some of the concerns suggested by the lack of previous research into how the relationships between timbral adjectives and the sounds they describe are formed. Similarly, to avoid other pitfalls of previous research, no initial assumptions are to be made about the words to be used such as pairings of adjectives implying opposition. As described in the previous section, a number of previous studies have gathered a variety of timbral adjectives from musicians. A pilot listening experiment will provide a useful means of assessing a selection of these words for likely usefulness, and an opportunity to gather other adjectives. This experiment will take the form of a listening experiment in which subjects rate samples against scales labelled with adjectives, with the opportunity to suggest other adjectives. A larger listening experiment will then be undertaken with a revised set of adjectives based on the results of the pilot experiment. The results of this will be used to inform selection of a set of timbral adjectives useful for controlling synthesis. That ultimate purpose dictates several conditions with which the resulting set of adjectives must comply. They should have a common meaning for all intended users, to permit the establishment of consistent relationships between timbral adjectives and synthesis parameters. The adjectives should be useful discriminators between a variety of sounds across the timbre space being studied. Finally, the descriptors should not be highly correlated with other descriptors, as this would lead to confusion in the establishing of timbral cues for particular adjectives. METHODS Initial test design was based on the psychoacoustic test procedure of Disley (2004), where subjects listen to ten auditory stimuli, rating each one on ten adjective scales. The descriptors were selected for this purpose by choosing words previously studied or summarised in Creasey (1998), Disley and Howard (2004), Moravec and Stepánek (2003) and Kendall and Carterette (1993, (1) and (2)). Adjectives with ambiguous or similar meanings were rejected (e.g. brilliant was considered too similar to bright to justify the inclusion of both in a small subset of timbral adjectives). The chosen adjectives were: bright, clear, warm, thin, flutey, harsh, dull, nasal, metallic and colourful. Each adjective scale was marked with eleven possible answer positions from, for example, not bright to bright. The sounds used were chosen to represent the range of Western orchestral instruments and synthesised sounds as adequately as possible within the constraints of only ten samples. As a widely available attempt to provide a comprehensive set of timbres already exists in General MIDI, a subset of GM patches was chosen from those available on a Yamaha XG module: piano, xylophone, bell, taiko drum, flute, oboe, viola, brass, sawtooth and sinewave. Samples were carefully chosen and adjusted to have no vibrato or use of multiple or stereo voices, and were recorded in mono at 44.1kHz, with the note g3 (MIDI note 67 or 392Hz) being held for 1.5 seconds. Sixteen musically trained listeners took part in the pilot experiment. y was demonstrated to be a poor discriminator, with most sounds being definitely not flutey apart from the flute and, to a much lesser extent, the sinewave. Thus although it accurately identifies the flute, its wider use is demonstrated to be limited. Disley (2004) hypothesized that flutey and dull were synonymous in the context of the pipe organ, but they are not significantly correlated here. The bell was considered particularly metallic, but elsewhere that word appeared a useful discriminator. For each application of an adjective rating scale, listeners were asked to indicate their confidence in that rating on a five point scale. Their confidence ratings were on average similar for every adjective, with the exception of colourful ISBN ICMPC 62

3 which had significantly lower confidence, suggesting that listeners are less sure of how to apply it in this context. Subjects suggested the following list of words as alternative timbral descriptors. Numbers in brackets refer to the number of different subjects who suggested that word, with words only suggested by one listener being excluded. percussive (11), reedy (6), pure (5), synthetic (5), wooden (5), buzzy, electronic, and stringy (all 4), bell-like, brassy, drum-like, gentle, plonky and resonant (all 3), blown, booming, chiffy, cold, evolving, plain, rich, ringing, strident, trumpet-like and ugly (all 2). Some of these words are emotive, synonymous or only apply to particular instrument families. Some (e.g. full) have been used in previous studies and rejected due to their lack of common understanding, but others are potentially useful. There is clearly a desire for scales to describe the onset, development and decay of the sound. Of the ten words used in the pilot experiment, flutey and colourful have been demonstrated as unsuitable for general use in timbral description. Listener comments suggested a test with more adjectives would not be too arduous, therefore in addition to the eight remaining words from the pilot study, the following seven words chosen from those suggested above will be used in the main experiment: wooden, rich, gentle, ringing, pure, percussive and evolving. Words which apply to only certain families of instruments (such as reedy) have been excluded, as have those that are too emotive or otherwise restrictive. With some of these words (evolving, for example), it is difficult to predict a direct relationship with any timbral quality, but rather than exclude all potentially difficult words, some have been included to examine how listeners use them and whether they are indeed useful. Many subjects commented on the artificial nature of the sounds used in this experiment, and in one case a subject noted that he had rated the sound as much more harsh than he would have done for their real-world equivalents. The larger scale experiment must therefore use recordings of real instruments. The use of real samples brings up a number of additional issues. Some instruments naturally incorporate vibrato, something avoided in the pilot test. Vibrato is an inherent part of some sounds; for voice and violin no vibrato at all encourages fission (hearing out of spectral components) rather than fusion, and can have a grating effect (Lerdahl, 1987). Hence if we are using real-world samples, vibrato will be an inevitable part of that. The use of real samples also brings with it the question of whether any synthesised samples should be used. Synthesis has an essentially unlimited tonal palette, and we have either the option of using synthesis primitives (e.g. sine, square and sawtooth waveforms) or choosing from an unlimited spectrum of sounds. The first option is problematic if real samples are to be used as well, as the raw synthesised sounds will inevitably seem crude in comparison with the sophisticated and complex real-world samples. The second option s openendedness creates problems, as most musical sounds are based on imitations of musical instruments. Although this is more a criticism of the limitations people typically impose upon synthesis than a criticism of synthesised sound in general, these difficulties suggest that synthesised sounds should be avoided at this point, and exclusively real-world samples (i.e. those generated by acoustic or electro-acoustic instruments) be used. Large-scale Experiment Based on the results of the pilot experiment, a larger scale listening experiment was designed along similar lines. Twelve instrument samples were chosen from the MUMS (McGill University Master Samples) library to represent the four broad groups of stringed instruments, brass instruments, woodwind instruments and percussive instruments. The precise samples chosen depended on the quality of sample for that particular note. Some samples had atypical overtones, onsets or other factors for the note chosen, and that is why, for example, a viola was used instead of a violin or cello. The samples chosen were: Strings: Viola Bowed (*), Viola Pizzicato, Electric Guitar Brass: Tenor, Bach Trumpet, Trumpet Harmon Woodwind: Vibrato (*), Alto Saxophone, Percussion: Hamburg Steinway, Tubular Bells, A Clarinet (E flat) recording was chosen for use as the initial demonstration sample. The note chosen for all thirteen samples was G=392Hz. Samples marked (*) included a large amount of vibrato. The electric guitar (without any effects) offered a distinctly different timbre within the string family. Among the brass family, the samples are respectively soft, with few overtones (trombone), bright (Bach trumpet), and extremely bright a Harmon muted trumpet is a distinctive and different sound, and offers the possibility of exploring greater timbral limits. In all cases, the samples were chosen to give the greatest variety of timbre within the constraints of twelve orchestral instrument samples and thus make the timbral space explored by this investigation as large as possible. Each listener rated twelve samples of musical instruments on fifteen scales. The scales were labelled with the timbral adjectives derived in the previous section, namely bright, clear, warm, thin, harsh, dull, nasal, metallic, wooden, rich, gentle, ringing, pure, percussive and evolving. Each scale again had a confidence scale so that listeners could indicate their confidence in their application of that word to that sound. Two groups of subjects were involved, one of 23 listeners who took the test ten times and will be referred to as the control group, and a larger group of 36 who took the test once. The control group took the listening test five times in a controlled environment, and in between those tests five ISBN ICMPC 63

4 times in an uncontrolled environment, both to explore the significance of a controlled environment and to assess the reliability of judgments in an initial test by looking at consistency of ratings over multiple tests. The larger group of subjects took the test in an uncontrolled environment. All listeners physically took the test by loading a webpage on a computer. The test had been designed such that the computer type and operating system were not important, the only requirements being an Internet connected computer with a sound card and a good quality pair of headphones. Listeners in uncontrolled situations were asked to ensure that they were not distracted by external noise or other interference. They were also asked to set the volume to a comfortable level using the demonstration sample and then not to alter it during the course of the test. Listeners in the controlled situation all used the same pair of headphones (Sennheiser EH2270) set at the same volume level. The controlled environment was simply a small room with a computer. The control group were not disturbed during the course of the test, but were free to leave the room and seek assistance if any problems or questions arose. Ideally in any such test the samples should be presented in random order. However this presents practical problems for both the testing software used to store the results, and for the control group as different answers between their multiple instances of test taking could be due to the differences in order of sample presentation. Four different test orders were therefore created, carefully chosen so that each sample didn t appear in the same position twice, and that no two samples appeared in the same order across the four tests. Each listener in the control group took the same test each time, but overall the effect should be that of a random test order, as any differences due to the sample order will cancel out when all results are combined, as almost equal numbers of listeners took each of the four versions. Listener Demographics The control group consisted of 23 adults, between the ages of 18 and people took part in the single experiment, aged between 18 and 27. All subjects spoke UK English as their first language, and all were students or staff in the subjects of Music and Music Technology at a UK University. For general analysis of results, it is justified to combine the results of the control group s first taking of the test with the results from the other listeners, provided that subsequent analysis does not reveal significant differences between the controlled and uncontrolled results. This gives a set of results for general analysis from a group of 59 UK English speakers, of whom 29 are female and 30 are male. RESULTS The average results from the group of 59 listeners can be arranged to show each instrument s rating for a single adjective, shown in figures 1 to 15 below. This shows clearly which adjectives were useful in discriminating between the samples. There is no vertical scale in these graphs samples are spaced out to aid clarity. The listeners appear to make good use of the possible range of the adjective scales in most cases. Pure has a poor spread of results, with all instruments gathered around the middle, suggesting that as a timbral adjective it offers poor discrimination between the samples of this timbre space. Rich has a similarly poor spread. Percussive appears to be used to divide the samples into two groups based largely on their amplitude envelopes. not bright bright Figure 1. Average results for the word bright not clear clear Figure 2. Average results for the word clear not warm warm Figure 3. Average results for the word warm ISBN ICMPC 64

5 not thin thin Figure 4. Average results for the word thin not nasal nasal Figure 8. Average results for the word nasal not wooden wooden Figure 5. Average results for wooden not metallic metallic Figure 9. Average results for metallic not harsh harsh Figure 6. Average results for the word harsh not pure pure Figure 10. Average results for the word pure not dull dull Figure 7. Average results for the word dull not percussive percussive Figure 11. Average results for percussive ISBN ICMPC 65

6 not rich rich Figure 12. Average results for the word rich not ringing Figure 14. Average results for ringing ringing not gentle Figure 13. Average results for gentle gentle It is possible that where there isn t a good spread on these scales, the timbre space described by the samples chosen simply does not represent the extremes that these adjectives could describe. For example, although none of these samples are not strongly not clear does not mean that clear is a poor timbral descriptor. These average results need to be combined with further analyses to understand which words are genuinely poor descriptors. The average standard deviation for each word gives a crude indication of agreement in usage of that word, with a high standard deviation corresponding to increased disagreement between listeners. The highest standard deviations, and therefore the least agreement on usage, are found for metallic, wooden and evolving. This suggests that these words are inappropriate for further study in this context, as without a common usage they cannot be intuitively and consistently understood by all musicians. Nasal and ringing had the next highest standard deviations, calling into question their common understanding among listeners. Comparison of results from the controlled and uncontrolled tests for the control group suggested that listening outside of the controlled situation had very little influence on subject answers. The most marked feature was a slight increase in the average confidence rating in the uncontrolled situation, but this was to a slight and statistically insignificant extent. Listeners were also slightly more likely to rate the words higher in an uncontrolled situation, although this not evolving evolving Figure 15. Average results for evolving was again to a relatively small and insignificant extent. With that caveat, listeners in a controlled environment are likely to rate samples such as those used here as less clear, less warm, more harsh, less pure and less gentle, but again none of these were to a statistically significant extent. As the group of 59 listeners consisted of nearly the same number of men and women, it is interesting to compare the results when divided by sex. It is impossible to attribute the differences directly to sex, but the results make interesting reading. The average male, for example, thinks a xylophone more wooden than the average female. On average, the female standard deviations were slightly higher than the male, indicating less agreement, but the bulk of this difference was for the term percussive, on which the males were notably more in agreement. Confidence levels were also largely similar, with the males having a slight peak in confidence for percussive. Male and female subjects were otherwise largely in agreement. Examination of listener confidence levels show a distinct dip in confidence for evolving, with all other words having much closer average confidence levels, the highest confidence ratings being given to clear, percussive, ringing and bright. Subjects were least confident in applying evolving to the Bach trumpet sample, and most confident in applying ringing and metallic to the tubular bell sample. While control group listeners taking the test multiple times showed little difference in the consistency of their results, their ISBN ICMPC 66

7 ICMPC9 International Conference on Music Perception and Cognition - Proceedings confidence levels did increase over the course of the ten tests. Correlation measures the degree to which variables move in similar (or opposite) ways. It is therefore useful in determining which adjectives may be synonymous or antonymous. This in turn would suggest which words might be removed on grounds of duplication. Unfortunately the large number of listeners results in a large number of adjective pairs having significant correlation. Significant opposition is demonstrated between bright and dull, warm with thin, harsh with gentle and wooden with metallic. However correlation can be misleading as it does not inherently take account of the scale of similarity; two variables that move in a similar pattern but to different degrees will show strong correlation, which explains why several of the adjectives which didn t discriminate well between the samples still showed significant correlation with other adjectives. Many of the correlations are undoubtedly important. However, just because two adjectives may be strongly correlated does not alone mean that they are synonymous. It could be, for example, that all the percussive sounds in the sample set chosen were also ringing, but that not all percussive sounds are necessarily also ringing. One solution to the problem of too many significant correlations is to use cluster analysis to give a measure of the relative similarity between multiple variables. Figure 16 shows a hierarchical cluster analysis for all fifteen timbral adjectives. Figure 16. Hierarchical cluster analysis of adjectives, using average linkage between groups This suggests that bright and clear, and warm and gentle are, within the context of the samples chosen, similar in application. Principal component analysis (PCA) gives an idea of how many underlying dimensions there are to the data. Again using all fifteen timbral adjectives as variables, PCA suggests that 91.7% of the variance can be explained by four principal components. The rotated versions of these are shown in table 1 below. Table 1. Principal components (Varimax rotated with Kaiser normalization, converged in 5 iterations) Component Bright Clear Warm Thin Wooden Harsh Dull Nasal Metallic Pure percussive Rich Gentle Ringing Evolving If those components were described in terms of the original timbral adjectives, the first one might scale from bright, thin and harsh to dull, warm and gentle. The combination of pure and percussive in the second component is interesting, in opposition to nasal. The third component is largely accounted for by metallic in opposition to wooden. The fourth scale is mainly evolving. Subjects made a number of interesting comments on test procedure. One suggested that differentiation was needed between pure in terms of pitch or sonority. Some subjects commented that evolving was hard to apply with such short samples, and several commented that they didn t feel they understood what it meant. Several subjects noted that their use of metallic and wooden frequently described the material of the instrument rather than its sound. Many subjects reported an element of applying ratings to sounds based on how they typically expected them to sound, rather than how that particular example sounded. This use of mental primitives may be worth bearing in mind when attempting to relate these adjectives. One listener suggested repeating the test with entirely synthesised sounds not based on any instruments, to get away from the problem of primitives. This concept of primitives may be important to further analysis as at least two subjects mistook the electric guitar sample for a harpsichord. Many subjects thought that the samples were artificial in nature, presumably due to them being presented in mono and without reverberation. Some instrumentalists thought that some samples were not played well, but that could be again due to the close-miking in an anechoic situation used by the ISBN ICMPC 67

8 ICMPC9 International Conference on Music Perception and Cognition - Proceedings MUMS samples. One subject commented that the note chosen used no valves on a trumpet, and thus the result would be purer than another note. However as any acoustic analysis will be done on these samples alone, the choice of note shouldn t affect this or any other sample in this manner. In response to a specific question, none of the subjects reported that they had synesthaesia, or thought key colour had an effect on their answers. Some thought that raising or lowering the pitch of the note would inevitably affect how bright, harsh and gentle it was perceived to be. CONCLUSIONS The analyses presented in the previous section suggest that considerable reduction of the number of adjective scales is possible without adversely reducing the possibility of describing sounds of the type use in this experiment. Obvious candidates for removal include those for which listeners were not in agreement (evolving, wooden and metallic), although these words may represent important general categories of description that require a different approach from that used here. Pure and rich are also candidates for rejection due to their poor discrimination between a varied selection of sounds, although it is possible that they and other timbral adjectives may be of more use in comparisons between similar sounds. Bright and dull are demonstrated to be in opposition, but the establishment of more definite adjective pairs is compromised by the high correlation between many adjectives. Thus although a set of ten adjectives are proposed at this stage for use in synthesis control (bright, clear, warm, thin, harsh, dull, nasal, gentle, ringing, and percussive) there is scope for further reduction. Nasal and ringing have less agreement between listeners than the other words in that selection. Future work will include looking for correlations between some of those adjectives and acoustic features, and initial attempts at this are described in Disley, Howard and Hunt (2006). It would also be informative to examine the principal components of the reduced adjective set, and look for correlation between those principal components and spectral and time domain features. Percussive, if it is to be used as an adjective referring to timbre, rather than amplitude, needs its amplitude and spectral cues isolating. REFERENCES Berger, K.W., (1965). Some factors in the recognition of timbre, Journal of the Acoustical Society of America, 36, (2), von Bismarck, G., (1974). Timbre of steady sounds: a factorial investigation of its verbal attributes, Acustica, 30, (3), Creasey, D.P., (1998). An exploration of sound timbre using perceptual and time-varying frequency spectrum techniques. DPhil thesis, The University of York. Disley, A.C., (2004). An exploration of timbral semantics related to the pipe organ. PhD thesis, The University of York. Disley, A.C. and Howard, D.M., (2003). Timbral semantics and the pipe organ, Proceedings of the Stockholm Music Acoustics Conference, (2), , KTH, Stockholm. Disley, A.C. and Howard, D.M., (2004). Spectral correlates of timbral semantics relating to the pipe organ, Proceedings of the Baltic-Nordic Acoustics Meeting, HUT, Helsinki. Disley, A.C., Howard, D. M. and Hunt, A. D., (2006). Musicians use of timbral adjectives, Proceedings of the Institute of Acoustics, 28, (1), Grey, J.M., (1975). An Explanation of Musical Timbre, PhD thesis, Stanford University. Howard, D.H. and Angus, J.A.S., (2001). Acoustics and Psychoacoustics, 2 nd edition, Oxford, Focal Press. Kendall, R.A. and Carterette, E.C., (1993 (1)). Verbal Attributes of Simultaneous Wind Instrument Timbres: I. von Bismarck s Adjectives, Music Perception, 10, (4), Kendall, R.A. and Carterette, E.C., (1993 (2)). Verbal Attributes of Simultaneous Wind Instrument Timbres: II. Adjectives Induced from Piston s Orchestration, Music Perception, 10, (4), Lerdahl, F., (1987). Timbral Hierachies, Contemporary Music Review, 2, Moravec, O. and Štĕpánek, J., (2003). Verbal description of musical sound timbre in Czech language, Proceedings of the Stockholm Music Acoustic Conference, 2, Nykänen, A. and Johansson, Ö., (2003). Development of a language for specifying saxophone timbre, Proceedings of the Stockholm Music Acoustics Conference, 2, Sandell, G. J., (1991). Aconcurrent Timbres in Orchestration: A Perceptual Study of Factors Determining Blend, PhD dissertation, School of Music, Northwestern University. ISBN ICMPC 68