A New Method for Traking Modulations in Tonal Musi in Audio Data Format 1 Hendrik Purwins, Benjamin Blankertz, and Klaus Obermayer CCRMA, Stanford Tehnial University Berlin, FR 2-1, FB 13, Franklinstr. 28/29, 10587 Berlin, Germany, hendrik,oby}@s.tu-berlin.de GMD FIRST, Rudower Chaussee 5, 12489 Berlin, Germany, blanker@first.gmd.de Abstrat Cq-profiles are 12-dimensional vetors, eah omponent referring to a pith lass. They an be employed to represent keys. Cq-profiles are alulated with the onstant Q filter bank [4]. They have the following advantages: (i) They orrespond to probe tone ratings. (ii) Calulation is possible in real-time. (iii) Stability is obtained with respet to sound quality. (iv) They are transposable. By using the q-profile tehnique as a simple auditory model in ombination with the SOM [11] an arrangement of keys emerges, that resembles results from psyhologial experiments [13], and from musi theory [1]. Cq-profiles are reliably applied to modulation traking by introduing a speial distane measure. Introdution The goal of this work is to derive an appropriate representation of tone enters based on the audio signal. To what degree does suh a representation have some psyhologial plausibility? Suh a method should be fast in alulation. It should be appliable for stylisti analysis and for tone enter traking. An interesting question is how far one an get just employing DSP, without deeper musial onsiderations. The probe tone experiments were pursued by Carol Krumhansl and Roger Shepard [14]. Probe tone ratings are a quantitative desription of a key, that reates the possibility of relating statistial or omputational analysis of musi to ognitive psyhology. The probe tone experiment onsists of two stages: establishment of a tonal ontext, and rating of the relation of a probe tone to that ontext. The tonal ontext is provided by examples, whih are unambiguously written in a ertain key. In our ase the subjets listen to simple adential hord progressions omposed of Shepard tones [13]: IV-V-I, VI-V-I, II-V-I (Roman numerals indiating sale degrees of the root of the hords). Subsequently, a Shepard tone hosen randomly from the hromati sale, the probe tone, is played. The subjet is asked to judge, how well the note fits with the tonal ontext, provided by the adential hord progression. The test subjets rate by a number from 1 ( fits poorly ) to 7 ( fits well ). After this proedure is repeated several times, with different hromati notes, the average rating for eah pith lass is alulated. The 12-dimensional vetor ontaining the averaged answers for eah pith lass is alled the probe tone rating. There are two types of rating vetors, one for major and one for minor depending on the mode of the ontexts. Rating vetors of keys in the same mode but with different toni keynotes are assumed to be related by a shift that ompensates for the interval of transposition (f. [13], p. 342). ontext /d d d /e e f f /g g g /a a a /b b -major -minor One observes, that the first sale degree is rated highest. The third and fifth sale degrees are also rated high. Diatoni notes are rated higher than non-diatoni notes. Aording to an observation reported in [12] (p. 66 76), eah omponent in the probe tone rating vetor orresponds to 1 H. Purwins, B. Blankertz, and K. Obermayer. A new method for traking modulations in tonal musi in audio data format. In International Joint Conferene on Neural Network (IJCNN 00) 6:270-275. IEEE Computer Soiety, 2000.
Purwins, Blankertz, and Obermayer. A new method for traking modulations... 271 the frequeny and the overall duration of ourrene of the orresponding pith lass at metrially prominent positions in a tonal piee that is written in a given key. Key distanes are alulated by omparing the orresponding probe tone ratings by orrelation, Eulidean distane, et. Our goal is different from pith reognition. We need not to know all exat pithes, just a profile whih indiates the key, resp. the tone enter. To see how a piee is represented, we have to onsider, how a note is represented. We will restrit us to a representation in a 12-dimensional vetor. Eah omponent in the vetor orresponds to a pith lass in the well tempered hromati sale. There are some approahes for automati tone enter reognition. Gang and Berger [7] introdued a system, based on input in midi data format. Linking metrial and harmoni information a reurrent net learns to make harmoni preditions. Griffith [8] did tone enter analysis on the simplest representation and referred to profiles that inluded interval use from eah pith lass [5]. Fujishima [6] mathes hords form the audio signal with some prototype hords based on Fourier tehniques. Leman [15] did tone enter analysis on the basis of referenes from Shepard tone adential hord progressions, whih where preproessed by an auditory model. Izmirli and Bilgen [10] used the onstant Q transformation [2] for tone enter analysis in ombination with a refined frequeny estimation observing phase hanges [3]. Context is integrated adaptively based on hord hanges. By anelling out harmonis of a deteted fundamental, fundamentals of other tones are possibly anelled out also. This method yields a quite reasonable, yet not perfet tone enter analysis. First we will introdue the onstant Q profile tehnique, and we will indiate its orrelation to the psyhologial probe tone data. Then we will show results in using the onstant Q profile tehnique as a simple auditory model in ombination with the Self Organizing Feature Map [11]. An arrangement of keys, whih is pereptually relevant, evolves on the basis of the Préludes Op. 28 by Chopin. Then a speial distane measure, the fuzzy distane, is introdued. It leads to good results in traking of modulations aross different tone enters. Cq-Profiles Cq-profiles are a new onept of key profiles. They unite features of Krumhansl s probe tone ratings [12] and Leman s orrelograms [15]. Advantages omprise: (1) Eah q-profile has a simple interpretation, sine it is a 12 dimensional vetor like a probe tone rating. The value of eah omponent orresponds to a pith lass. (2) A q-profile an easily be alulated from an audio reording. Sine no omplex auditory model, or other time onsuming method is used, the alulation is quik and an be done in real time. (3) The alulation of the q-profiles is very stable with respet to sound quality. E.g. analyzing a reording of Alfred Cortot from 1933/34 works well. Constant Q transform The alulation of the q-profiles is based on the onstant Q transform [2]. The letter Q refers to the onstant quotient of enter frequeny and bandwidth for eah filter. The onstant Q transform is useful in establishing a diret orrespondene between filters and musial notes by identifying appropriate enter frequenies. To minimize spetral leakage (f. [9]), we use 36 filters per otave rather than 12. Figure 1 shows how q-profiles are alulated from the output of the transform. Hene, only every third filter output maps to a tone of the hromati sale. Cq-profiles an be used to study pith use in different omposers and for modulation traking. A q-referene set is a sequene of 24 qprofiles, one for eah key. Every profile should reflet the tonal hierarhy that is harateristi for its key. Typially q-referene sets are alulated from sampled adential hord progressions or from small piees of musi. Calulation of q-transform Like the Fourier transform, a onstant Q transform [2] is a bank of filters, but in ontrast to the former it has geometrially spaed enter frequenies and a onstant ratio of frequeny to bandwidth q (! ), where " ditates the number of filters per otave.
Purwins, Blankertz, and Obermayer. A new method for traking modulations... 272 This is ahieved by hoosing an appropriate window length individually for eah omponent of the onstant Q transform (q-bin). For integer values the -th q-bin is the -th DFT-bin with window length. Calulation: First hoose minimal frequeny and the number of bins per otave " aording to the requirements of the appliation and let 2 max : ", 10243,! and (for ). Then the th q-bin is equal to "$# %'&)( # %'&+* -,/.. Following [4] we use Hamming windows 5 ( # %'& % 76. Using Parseval s rule a filter matrix is alulated in advane. Exploiting sparsity aelerates the alulation of the onstant Q transform very muh [4]. 45 40 35 30 db 25 20 15 10 5 0 #debe f f# gababbb #debe f f# gababbb #debe f f# gababbb #debe f f# gababbb d e f e f a b g a b FIGURE 1: The onstant Q transform is alulated from a minor third e (played on piano) with three bins per half-tone (left figure). We yield the onstant Q profile (right figure) by summing up bins for eah tone over all otaves. Appliations Derivation of a Toroidal Model of Inter-Key Relations (ToMIR) with q-referene sets In an experiment with musi in audio data format we ombine the q-transform and a toroidal Self Organizing Feature Map (SOM, [11]). After training the evolved onfiguration resembles ToMIR in musi psyhology ([12] p. 46) and musi theory ([16] pp. 19). The irles of fifths wrap around the torus in three turns. On the onfiguration, keys share borders with the dominant, the subdominant, the parallel and the relative key, f. figure 11 in [1]. Comparison of different hierarhies An important question that arises regarding q-referene sets that are alulated from audio reordings is in what respet the results are affeted by (1) musial interpretation, (2) the reorded instrument, and (3) the seleted piees of musi. For the examination of (1) we ompared radially different interpretations of the Chopin préludes and Bah WTC I preludia. The mean profiles showed a orrelation of 3 0.995/0.989. For the investigation of (2) we ompared the reordings of the preludia of Bah s WTC I preformed on modern pianos and on a (Pleyel) harpsihord: The orrelation is 0.989/0.982. To study the impat of the seletion of musi (3) on the orresponding referene sets, we performed some inter and some aross epoh omparisons. Group 1 onsists of four referene sets alulated from the preludia/fugues yles (separately) of both books of the well-tempered lavier (Glenn Gould s reording). Group 2 onsists of two referene sets derived from Alfred Cortot s reording of the Chopin préludes op. 28 (finished 1839), and from Olli Mustonen s reording of Alkan s préludes op. 31 (finished 1847). Group 3 onsists of a referene set based on Sriabin s preludes op. 11 (finished 1896) performed by Vladimir Sofronitsky, Heinrih Neuhaus, Vladimir Horowitz and the omposer 28-9: denotes the least integer greater than or equal to 9. 3 When writing orrelation values in the form 9 /; we use the onvention that 9 refers to major profiles and ; to minor profiles.
Purwins, Blankertz, and Obermayer. A new method for traking modulations... 273 q profiles of adenes spetrally weighted ratings q profiles of adenes spetrally weighted ratings C C# D D# E F F# G G# A A# B (a) Major ratings C C# D D# E F F# G G# A A# B (b) Minor ratings FIGURE 2: The q-profiles of sampled piano adenes are ompared with spetrally weighted ratings. (reprodued from a Welte-Mignon piano roll). The inter-group orrelations are 0.992/0.983 for the Bah referene sets (mean value) and 0.987/0.980 between Chopin s and Alkan s préludes. The mean aross group orrelations are 0.924/0.945 between groups 1 and 2, 0.935/0.949 between groups 1 and 3 and 0.984/0.952 between groups 2 and 3. Relating q-profiles to probe tone ratings Krumhansl observed in [12] a remarkable orrelation between the probe tone ratings and the total ourrenes of the twelve hromati sale tones in musial ompositions. In order to establish a diret orrespondene between probe tone ratings and profiles of a q-referene set, one fat has to be taken into onsideration. In the q-profiles not only the played tones are registrated, but all harmonis. For piano tones the strongest frequeny ontribution falls (modulo otaves) on the toni and on the dominant keynote in an approximate average ratio 3:1. Hene q-profiles should not be ompared with the probe tone ratings, but with adapted ratings, in whih the harmoni spetrum of the analyzed tones is aounted for. Suh a spetrally weighted rating is alulated by adding to the rating value for eah tone one third of the rating value for the tone seven hromati steps above (modulo otave). Figure 2 shows the orrelation of the qprofiles of sampled piano adenes (I-IV-V 7 -I and I-VI-V 7 -I) with the spetrally weighted ratings. Appliation in tone enter traking How an a piee be lassified aording to a q-referene set? Generally we have the problem of mathing a given q-profile with a profile of the q-referene set. A typial mathing riteria is the losest fuzzy distane: Let be a value subjet to an unertainty quantisized by a value (typially is the mean and the standard deviation of some statistial data). The fuzzy distane of some value " to regarding is defined by " " *. The fuzzy distane is similar to the Eulidean metri, but the greater the unertainty the more relaxed is the metri. As an example, we present an analysis of Chopin s -minor Prélude op. 28, No. 20. The referene vetors were alulated from all 24 Chopin Préludes in audio format. In the sore (Figure 3 (b)) tone enters are marked. They were determined by a musial expert. Tone enters in parentheses indiate toniizations on a very short time sale. Sine the automati tone enter reognition (Figure 3 (a)) does not look ahead, there is a delay in reognizing tone enters. The program aptures the prevailing key minor and the modulations: minor (1.measure), major (2.measure). 4 In measure 3, beause of the interdominants (1.beat) and (2.beat) there is a short toniization for minor and for f minor. Then C major is indiated (beat 4). Measure 4 shows G-major. In measures 5 and 6 a faux bordun in minor ours, on a oarse time sale. Short toniizations our in measure 5, beat 4 (g minor) and measure 6, beat 3 (G major). In measure 7, there is a lear 4 In measure 3 on beat 3 there is an!#". [17] points out this being a typo in Chopin s manusript. He argues for replaing!#" by!#$ ", beause major and minor keys should alternate within the first 4 measures. We analyze an unrevised version.
Purwins, Blankertz, and Obermayer. A new method for traking modulations... 274 Key analysis, minor=grey, major=blak C Db D Eb E F Gb G Ab A Bb B Time (a) Result of automati tone enter analysis 5 A () (f) (C) G 7 10 A A (b) Sore FIGURE 3: Chopin s -minor prélude, op. 28, No. 20. In (a) grey indiates minor, blak indiates major. If there is neither blak nor grey at a ertain time, the signifiane of a partiular key is below a given threshold. There is no distintion between enharmoni equivalent keys. adene in minor. In measure 8 beat 1 and 2 we have a flavour of Major or Major. The analysis indiates Major. In measure 8 beat 4, the piee returns to minor. The analysis indiates this with a delay, beause in the performane of Cortot Major is heavily emphasized. Measures 9-12 are the same as 5-8, exept the level is pp now. Therefore the analysis is more unertain. This result is astonishing. The only expliit musial knowledge utilized is the display of the signal in terms of pith lasses. The system reeives musial knowledge only by hoie of the musi piees, whih lead to the referene vetors.
Purwins, Blankertz, and Obermayer. A new method for traking modulations... 275 Disussion The simple onstant Q profile method inorporates ontext proessing by averaging over the entire piee. Only very basi musi theoretial assumptions like otave equivalene and the hromati sale are used expliitly. However it an apture a large amount of harmoni struture inluding modulation and keys. Other musial knowledge is not expliitly used, like voie leading, harmony, metri, and rhythm. The onstant Q profile method is a powerful tool that an be extended to different tunings, and to real time analysis. It ould be improved by modeling masking phenomena. Appliations inlude automati modulation traking, and analysis of pith use in different omposers and epohs. A forthoming paper will over more details. We are grateful to H. P. Reutter, and A. Budde. The first author was supported by Studienstiftung des deutshen Volkes and Axel Springer Stiftung. Referenes [1] B. Blankertz, H. Purwins, and K. Obermayer. Toroidal models of inter-key relations in tonal musi. In VI. International Conferene on Systemati and Comparative Musiology, 1999. submitted. [2] J. Brown. Calulation of a onstant Q spetral transform. J. Aoust. So. Am., 89(1):425 434, 1991. [3] J. C. Brown and M. S. Pukette. A high resolution fundamental frequeny determination based on phase hanges of the Fourier transform. Journal of the Aoustial Soiety of Ameria, 1993. [4] Judith C. Brown and Miller S. Pukette. An effiient algorithm for the alulation of a onstant Q transform. J. Aoust. So. Am., 92(5):2698 2701, 1992. [5] R. Browne. Tonal impliations in the diatoni set. In Theory Only, 5:3 21, 1981. [6] T. Fujishima. Realtime hord reognition of musial sound: a system using Common Lisp Musi. In International Computer Musi Conferene, pages 464 467. ICMA, 1999. [7] D. Gang and J. Berger. A unified neurosymboli model of the mutual influene of memory, ontext and predition of time ordered sequential events during the audition of tonal musi. In Hybrid Systems and AI: Modeling, Analysis and Control of Disrete + Continuous Systems. AAAI Tehnial Report SS-99-05, 1999. [8] N. Griffith. Development of tonal enters and abstrat pith as ategorizations of pith use. In Connetion Siene, pages 155 176. MIT Press, Cambridge, 1994. [9] F. J. Harris. On the use of windows for harmoni analysis with disrete fourier transform. In Pro. IEEE, volume 66, pages 51 83, 1978. [10] Ö. Izmirli and S. Bilgen. A model for tonal ontext time ourse alulation from aoustial input. Journal of New Musi Researh, 25(3):276 288, 1996. [11] T. Kohonen. Self-organized formation of topologially orret feature maps. Biol. Cybern., 43:59 69, 1982. [12] C. Krumhansl. Cognitive Foundations of Musial Pith. Oxford University Press, Oxford, 1990. [13] C. L. Krumhansl and E. J. Kessler. Traing the dynami hanges in pereived tonal organization in a spatial representation of musial keys. Psyhologial Review, 89:334 68, 1982. [14] C. L. Krumhansl and R. N. Shepard. Quantifiation of the hierarhy of tonal funtion with a diatoni ontext. Journal of experimental psyhology: Human Pereption and Performane, 1979. [15] M. Leman. Shema-based tone enter reognition of musial signals. Journal of New Musi Researh, 23:169 204, 1994. [16] A. Shoenberg. Strutural funtions of harmony. Norton, 1969. [17] E. Zimmermann. Chopin Préludes op. 28: Kritisher Beriht. Henle, 1969.