Followng a muscal performance from a partally specfed score. Bryan Pardo and Wllam P. Brmngham Artfcal Intellgence Laboratory Electrcal Engneerng and Computer Scence Dept. and School of Musc The Unversty of Mchgan 110 ATL buldng, 1001 Beal Drve Ann Arbor, MI 48109 bryanp@umch.edu, wpb@eecs.umch.edu Abstract Ths paper descrbes a system that maps a muscal performance, recorded as MIDI, onto a partally specfed score (lead sheet). Our system breaks the performance nto approprate segments for hypotheszed chords by representng t as a drected acyclc graph (DAG) whose edges represent tme-ntervals of the musc. The hghest-reward path through the DAG corresponds to the best segmentaton of the performance. Durng segmentaton, the name of the best matchng chord for each segment label s gven to the segment. The sequence of chord names generated by the performance analyss s then algned aganst the sequence of chord names n the lead sheet usng an algnment technque drawn from gene-sequence analyss. Scores for the algnment are determned by estmatng the probablty of a random match between chord sequences compared to the probablty of the match. Probabltes are calculated through the analyss of chord frequences on a corpus of tonal musc drawn from a standard musc theory text. 1 Introducton Automated muscal accompanment that reacts naturally to the human performer s a longstandng goal of computer-musc research. Algorthms that match a wrtten score to a human performance are essental for an automated accompanst that reacts approprately durng performance. Systems that perform ths functon are called score matchers. Score-matchng research (Dannenberg 1984; Dannenberg 1988; Puckette and Lppe 1992; Vantomme 1995; Henk, Desan et al. 2000) has mostly concentrated on matchng a score that very closely specfes the ptch and orderng of every note n a pece of musc to the performance of a muscan attemptng to recreate the score as exactly as possble. Ths results n approaches whose goal s producng one-to-one mappng between score events and performance events. Many genres of folk and popular musc have partally specfed scores (lead sheets) that notate only man melodc notes, basc harmones and overall structure, supplyng far less muscal nformaton than a classcal score. Muscans necessarly mprovse melodc embellshments, the fguraton of the harmonc accompanment and even vary the overall form. Ths presents a problem for the tradtonal score-matchng approach: the score events wll be far fewer, and perhaps sgnfcantly dfferent, than the performance events. Therefore, a one-to-one mappng s not possble. 2001 IEEE 202 Proceedngs of MTAC 2001
Ths paper descrbes a system that maps a MIDI performance of a pece onto a lead sheet. We nduce a metrc reducton from the performance, removng structurally unmportant notes, leavng a rhythmcally smplfed muscal surface that emphaszes chord tones. Ths structure s automatcally annotated wth the assocated chord names. These chord labels are algned (matched) to the chords n the lead sheet usng a method borrowed from research n bologcalsequence algnment. 2 Fndng and Usng the Metrc Reducton Our approach to metrc reducton s based on automated chordal analyss. Our metrc-reducton system has a vocabulary of sx chord qualtes (maor, mnor, domnant 7, dmnshed, halfdmnshed, and fully dmnshed). Gven 12 possble roots for each qualty, ths gves a vocabulary of 72 chord templates. Durng processng, the system constructs a drected acyclc graph (DAG) whose edges represent tme-ntervals of the musc. The reward of an edge n the DAG corresponds to how well the notes n the correspondng segment match those n the closest chord template avalable to the system. The hghest-reward path n the DAG corresponds to the best segmentaton of the performance, whch can be computed n lnear tme wth respect to the number of chords (Pardo and Brmngham 2001). Once a performance s segmented, the best matchng chord template for each segment label s used to flter the performance for that segment. Notes matchng a chord element are consdered harmonc. Harmonc notes, when extended to the length of each segment, approxmate a metrc reducton of the musc. Fgure 1 contans a performance, a metrc reducton of the performance generated by our system, and the lead sheet upon whch the performance was based. Fgure 1 shows that the metrc reducton, whle closer to the lead sheet than the performance, has much detal not found n the lead sheet, leavng open the queston of what should be matched between lead sheet and metrc reducton. In ths paper, we focus on the matchng the sequence of chord labels n the metrc reducton to the sequence of chords n the lead sheet. Fgure 1: Lead sheet, performance, and metrc reducton 2001 IEEE 203 Proceedngs of MTAC 2001
2.1 Chord Sequence Algnment Once the problem has been cast as fndng the best algnment between two sequences drawn from the same alphabet of symbols (chord names), we can draw upon the extensve work done n the feld of bologcal sequence analyss. Dynamc-programmng-based approaches have been used wth success for over 30 years (Needleman and Wunsch 1970) to algn gene sequences based on a common ancestor. We have based our chord-algner on a dynamc-programmng algorthm descrbed by Gotoh (Gotoh 1982), whch s desgned to fnd the globally best algnment among sequences, gven some match scorng model. 2.2 Par Match Scores In comparng sequences, we look for evdence that each element n chord sequence A s related to some element n sequence B and then estmate the probablty that the sequences are related rather than a chance par. Here s a summary of our notaton. Let there be a par of chord sequences, A and B, of length A and B, respectvely. Let a be the th element of A and b be the th element of B. Let the elements of each sequence be drawn from the same alphabet. In our case, the alphabet contans the 72 chord names used for metrc reducton. Assume each element, e, of the alphabet occurs ndependently wth some known pror probablty, e). Then, the probablty of a random ont occurrence (R) of two sequences A and B s the product of the pror probabltes of the sequence elements. P ( A, B R) = a ) b ) (2.2.1) If, ndeed, the two sequences are a match (M) of some knd, then algned pars of chords occur wth a ont probablty that ndcates how lkely t s that the elements are related. P ( A, B M ) = a, b M ) (2.2.2) The odds rato s the rato between these two equatons. a, b M ) a, b M ) A, B M ) = f =, then = (2.2.3) A, B R) a ) b ) a ) b ) In order to have a smple addtve model, we take the log of ths rato, S = s( a, b ) (2.2.4) where a, b M ) s ( a, b) = log (2.2.5) a) b) s the log of the lkelhood of the par (a, b) occurrng because they are related as opposed to occurrng randomly. One nce feature of usng the log rato s that when the probablty of a match 2001 IEEE 204 Proceedngs of MTAC 2001
s below that of a random co-occurrence, the value s negatve. Smlarly, the value s postve when a match s more lkely than random chance. The values for s(a,b) can be placed n a score matrx, where the element, represents the score for the log lkelhood match of the th chord n the alphabet to the th chord n the alphabet. In order to fll the score matrx, one must fnd the pror probablty for each of the sx chord qualtes n all 12 keys. A natural way to fnd the pror probablty of each chord s to take the observatonal frequences of chord occurrence from a representatve corpus of musc. The Kostka-Payne Corpus s a set of 46 excerpts of tonal musc from varous perods. These excerpts have been annotated wth chord labels. The occurrence count for each of the sx chord qualtes used n our system was tabulated from these annotatons. The pror probablty for each of the 12 chords of a partcular qualty was then taken to be the probablty of the chord qualty dvded by 12. For example, 46.3% of the chord annotatons n the corpus were maor trads. Thus, C maor ) = 0.436/12 = 0.03633. Each s(a,b) also requres an estmate of a,b M). To fll n the full score matrx, ths must be done for all chord pars (a,b). Ths probablty can be thought of as the probablty that a segment of musc derved from chord a s assgned a label b by our metrc reducton system. To estmate ths probablty, we agan turned to the Kostka-Payne corpus. Each excerpt n the corpus was labeled wth chord names by our metrc reducton system and the resultng labels were compared to those gven n the annotated corpus. From ths, a table was compled where the element, represents the number of tmes our system labeled a segment wth the th chord n the alphabet and the corpus labeled t wth the th chord n the alphabet. The fnal score matrx was then derved from the pror probabltes and ths table, n accordance wth Equaton 2.2.5. 2.3 The Algnment Algorthm We use a dynamc-programmng algorthm ntroduced by Gotoh (1982), as descrbed n (Durbn, Eddy, et al. 1998) to fnd an optmal global algnment between two sequences A and B, allowng gaps between the sequences. Ths s done by constructng a matrx F, where F(,) s the score of the best algnment between the ntal segment a 1 through a of A and the ntal segment b 1 through b of B. The process s ntalzed by settng F(0,0) = 0. Thereafter, the elements of the matrx are flled n usng Equaton 2.3.1, where d s the gap penalty assgned to skppng an element of ether sequence to perform the algnment. The top lne n Equaton 2.3.1 gves the reward assgned n the score matrx for callng (a, b ) a match. The mddle lne calculates the penalty for skppng b and the lowest fnds the penalty for skppng a. F( 1, 1) + s( a, b ) F(, ) = max F( 1, ) d (2.3.1) F(, 1) d Note that, n the case of score followng, the lead sheet chord sequence s known n advance. Snce each cell n the array s flled n usng the values of the tems above and to the left of t, each row may be flled n usng only the knowledge of the pror row. Ths lets the system fll n ths table as the metrc reducton s beng created, wth a fxed number of steps per row. The tme 2001 IEEE 205 Proceedngs of MTAC 2001
to fll n the table s lnear wth respect to the length of the performance, so ths method presents no theoretcal lmt on the ablty of a system to score-match n real-tme. As F s flled n, another table must be kept, keepng track of the parent used to fll n each cell F(,). In many cases, t may be that more than one of the parents from Equaton 2.3.1 gves the maxmum value. In ths case, all parents are noted. Once ths s done, the best-scorng algnment may be read by startng at the fnal cell n the matrx and tracng backwards through the seres of parents used to generate the score. Table 1 shows the algnment score matrx, F, for the chord sequence from the lead sheet (columns) and metrc reducton (rows) n Fgure 1, gven a gap penalty, d = 1, and the probabltes calculated from the Kostka-Payne corpus. All maxmal scorng algnments are shown by paths of arrows through the table. In ths table, a vertcal arrow ndcates a skp of one chord n the metrc reducton sequence, a horzontal arrow s a skp of a chord n the lead sheet, and a dagonal arrow ndcates a match between the two. Table 1: Algnment scores for sequences from Fgure 1, gven d = 1. Lead sheet chord names are on the horzontal axs. Metrc reducton chord names are on the vertcal axs. G maor D dom7 G maor D dom7 G maor 0-1 -2-3 -4-5 G maor -1 6.24 5.24 4.24 3.24 2.24 F# dm -2 5.24 12.13 11.13 10.13 9.13 B mnor -3 4.24 11.13 14.84 14.24 13.24 D dom7-4 3.24 11.85 13.84 22.45 21.45 G maor -5 2.24 10.85 18.08 21.45 28.68 D dom7-6 1.24 9.85 17.08 25.7 27.68 E dom7-7 0.24 8.85 16.08 24.7 27.83 D maor -8-0.76 7.85 15.08 23.7 26.83 G maor -9-1.76 6.85 14.08 22.7 29.93 D dom7-10 -2.76 5.85 13.08 21.7 28.93 G maor -11-3.76 4.85 12.08 20.7 27.93 Note that, whle not shown n ths example, t s qute possble for the algnment matrx to show matches between chords of dfferent name f they have a hgh probablty of matchng due to chord smlarty and the gap penalty s large n comparson wth the match scores. For the purpose of score followng, Table 1 s used to determne the most probable place(s) n the lead sheet gven the current chord reported n the metrc reducton. For example, the nnth chord n the metrc reducton s G maor. The arrows n the table show that there are two best algnments for ths chord. The frst s to algn t wth the thrd chord n the lead sheet. The other s to algn t wth the fnal chord n the lead sheet. 2001 IEEE 206 Proceedngs of MTAC 2001
3 Conclusons We have descrbed a system that maps a MIDI performance of a pece onto a partally specfed score (lead sheet) for that pece. Ths s done by mappng the sequence of chords nduced from an automatcally generated metrc reducton of the performance onto the sequence of chords n the lead sheet. Both the metrc reducton and the algnment matrx can be generated n lnear tme wth respect to the length of the performance. Thus, ths approach s a good one for a real-tme matcher to a partally specfed score. Future work n ths area ncludes a more realstc gap penalty that reflects chord tmng rather than smply order, a hgher-level process to dsambguate between equal-scorng algnments and a real-tme mplementaton of a score-matchng system based on these technques. 4 References Dannenberg, R. (1984). "An On-Lne Algorthm for Real-Tme Accompanment." Internatonal Computer Musc Conference, Internatonal Computer Musc Assocaton. Dannenberg, R. B. (1988). "New Technques for Enhanced Qualty of Computer Accompanment." Proceedngs of the Internatonal Computer Musc Conference, Ann Arbor, MI. Durbn, R., S. Eddy, A. Krogh, G. Mtchson (1998). Bologcal Sequence Analyss, Probablstc models of protens and nuclec acds. Cambrdge Unversty Press, Cambrdge, U.K. Gotoh, 0. (1982). An mproved algorthm for matchng bologcal sequences Journal of Molecular Bology 162: 705-708. Henk, H., P. Desan, et al. (2000). Make Me a Match: An Evaluaton of Dfferent Approaches to Score-Performance Matchng. Computer Musc Journal 24(1): 43-46. Needleman, S. B. and C. D. Wunsch (1970). "A general method applcable to the search for smlartes n the amno acd sequence of two protens" Journal of Molecular Bology 48:443-453 Pardo, B. and W. Brmngham (2001). "The Chordal Analyss of Tonal Musc." Techncal report CSE-TR-439-01, The Unversty of Mchgan, Dept. of Electrcal Engneerng and Computer Scence, Ann Arbor, MI Puckette, M. and C. Lppe (1992). Score Followng In Practce. Internatonal Computer Musc Conference, Internatonal Computer Musc Assocaton. Vantomme, J. (1995). Score Followng by Temporal Pattern. Computer Musc Journal 19(3): 50-59. 2001 IEEE 207 Proceedngs of MTAC 2001