Music Plus One and Machine Learning

Transcription

1 Christoher Rahael School of Informatics and Comuting, Indiana University, Bloomington Abstract A system for musical accomaniment is resented in which a comuter-driven orchestra follows and learns from a soloist in a concerto-like setting. The system is decomosed into three modules: the first comutes a real-time score match using a hidden Markov model; the second generates the outut audio by hase-vocoding a reexisting audio recording; the third rovides a link between these two, by redicting future timing evolution using a Kalman filter-like model. Several examles are resented showing the system in action in diverse musical settings. Connections with machine learning are highlighted, showing current weaknesses and new ossible directions. 1. Musical Accomaniment Systems Musical accomaniment systems are comuter rograms that serve as musical artners for live musicians, usually laying a suorting role for music centering around the live layer. The tyes of ossible interaction between live layer and comuter are widely varied. Some aroaches create sound by rocessing the musician s audio, often driven by analysis of the audio content itself, erhas distorting, echoing, harmonizing, or commenting on the soloist s audio in largely redefined ways, (Lie, 2002), (Rowe, 1993). Other orientations are directed toward imrovisatory music, such as jazz, in which the comuter follows the outline of a score, erhas even comosing its own musical art on the fly (Dannenberg & Mont-Reynaud, 1987), or evolving as a call and resonse in which the comuter and human alternate the lead role (Franklin, 2002), (Pachet, 2004). Our focus here is on a third aroach that models the traditional classical concerto-tye setting in which the Aearing in Proceedings of the 27 th International Conference on Machine Learning, Haifa, Israel, Coyright 2010 by the author(s)/owner(s). comuter erforms a recomosed musical art in a way that follows a live soloist (Dannenberg & Mukaino, 1988),(Cont et al., 2005),(Rahael, 2009). This categorization is only meant to summarize some ast work, while acknowledging that there is considerable room for blending these scenarios, or working entirely outside this realm of ossibilities. The motivation for the concerto version of the roblem is strikingly evident in the Jacobs School of Music (JSoM) at Indiana University, where most of our recent exeriments have been erformed. For examle, the JSoM contains about 200 student ianists, for whom the concerto literature is central to their daily ractice and asirations. However, in the JSoM the regular orchestras erform only two iano concerti each year using student soloists, thus ensuring that most of these asiring ianists will never erform as orchestral soloist while at IU. We believe this is truly unfortunate since nearly all of these students have the necessary technical skills and musical deth to greatly benefit from the concerto exerience. Our work in musical accomaniment systems strives to bring this rewarding exerience to the music students, amateurs, and many others who would like to lay as orchestral soloist, though, for whatever reason, don t have the oortunity. Even within the realm of classical music, there are a number of ways to further subdivide the accomaniment roblem, requiring substantially different aroaches. The JSoM is home to a large string edagogy rogram beginning with students at 5 years of age. Students in this rogram lay solo ieces with iano even in their first year. When accomanying these early-stage musicians, the ianist s role is not simly to follow the young soloist, but to teach as well, by modeling good rhythm, steady temo where aroriate, while introducing musical ideas. In a sense, this is the hardest of all classical music accomaniment roblems, since the accomanist must be exected to know more than the soloist, thus dictating when the accomanist should follow, as well as when and how to lead. A coarse aroximation to this accomanist

2 role is to rovide a rather rigid accomaniment that is not overly resonsive to the soloist s interretation (or errors) there are several commercial rograms that take this aroach. The more sohisticated view of the edagogical music system one that follows and leads as aroriate is almost comletely untouched, ossibly due to the difficulty of modeling the objectives. However, we see this area as fertile for lasting research contributions and hoe that we, and others, will be able to contribute to this cause. An entirely different scenario deals with music that evolves largely without a sense of rhythmic flow, such as in some comositions of Penderecki, Xenakis, Boulez, Cage, and Stockhausen, to name some of the more famous examles. Such music is often notated in terms of seconds, rather than beats or measures, to emhasize the irrelevance of traditional ulse. For works of this tye involving soloist and accomaniment, the score can indicate oints of synchronicity, or time relations, between various oints in the solo and accomaniment arts. If the aroach is based solely on audio, a natural strategy is simly to wait until various solo events are detected, and then to resond to these events. This is the aroach taken by the IRCAM score follower, with some success in a variety of ieces of this tye (Cont et al., 2005). A third scenario, which includes our system, treats works for soloist and accomaniment having a continuing musical ulse, including the overwhelming majority of common ractice art music. This music is the rimary focus of most of our erformance-oriented music students at the JSoM, and is the music where our accomaniment system is most at home. Music containing a regular, though not rigid, ulse requires close synchronization between the solo and accomanying arts, as the overall result suffers greatly as this synchrony degrades. Our system is known interchangeably as the Informatics Philharmonic, or Music Plus One (MPO), due to its alleged imrovement on the lay-along accomaniment records from Music Minus One that insired our work. For several years we have been collaborating with faculty and students in the JSoM on this traditional concerto setting, in an ongoing effort to imrove the erformance of our system while exloring variations on this scenario. A video of such a collaboration is contained in htt:// edu/aers/icml10, while also roviding further examles relevant to our discussion here. We will resent a descrition of the overall architecture of our system in terms of its three basic comonents: Listen, Predict, and Play, including several illuminating examles. We also identify oen roblems or limitations of roosed aroaches that are likely to be interesting to the Machine Learning community, and well may benefit from their contributions. The basic technology required for common ractice classical music extends naturally to the avant garde domain. In fact, we believe one of the greatest otential contributions of the accomaniment system is in new music comosed secifically for human-comuter artnershis. The comuter offers essentially unlimited virtuosity in terms of laying fast notes and coordinating comlicated rhythms. On the other hand, at resent, the comuter is comaratively weak at roviding aesthetically satisfying musical interretations. Comositions that leverage the technical ability of the accomaniment system, while humanizing the erformance through the human soloist s leadershi, rovide a oen-ended musical meeting lace for the 21stcentury comosition and technology. Several comositions of this variety, written secifically for our accomaniment system by Swiss comoser and statistician Jan Beran, are resented at the web age referenced above. 2. Overview of Music Plus One Our system is comosed of three sub-tasks called Listen, Predict, and Play. The Listen module interrets the audio inut of the live soloist as it accumulates in real-time. In essence, Listen annotates the incoming audio with a running commentary, identifying note onsets with variable detection latency, using the hidden Markov model discussed in Section 3. A moment s thought here reveals that some detection latency is inevitable since a note must be heard for an instant before it can be identified. For this reason we believe it is hoeless to build a urely resonsive system one that waits until a solo note is detected before laying a synchronous accomaniment event: our detection latency is usually in the ms. range, enough to rove fatal if the accomaniment is consistently behind by this much. For this reason we model the timing of our accomaniment on the human musician, continually redicting future evolution, while modifying these redictions as more information becomes available. The module of our system that erforms this task, Predict, is a Gaussian grahical model quite close to a Kalman Filter, discussed in Section 4. The Play module uses hase-vocoding (Flanagan & Golden, 1966) to construct the orchestral audio outut using audio from an accomaniment-only recording. This well-known technique wars the timing of the

3 Note 1 Note 2 Note 3 etc. q q q q q q atck atck start 1 m start 2 start 3 Figure 1. The state grah for the hidden sequence, x 1, x 2,..., of our HMM. original audio without introducing itch distortions, thus retaining much of the original musical intent including balance, exression, and tone color. The Play rocess is driven by the outut of the Predict module, in essence by following an evolving sequence of future targets like a trail of breadcrumbs. While the basic methodology of the system relies on old standards from the ML community HMMs and Gaussian grahical models the comutational challenge of the system should not be underestimated, requiring accurate real-time two-way audio comutation in musical scenarios comlex enough to be of interest in a sohisticated musical community. The system was imlemented for off-the-shelf hardware in C and C++ over a eriod of about fifteen years by the author, resulting in more than 100,000 lines of code. Both Listen and Play are imlemented as searate threads which both make calls to the Predict module when either a solo note is detected (Listen) or an orchestra note is layed (Play). What follows is a more detailed look at Listen and Predict. 3. Listen: HMM-Based Score Following Blind music audio recognition treats the automatic transcrition of music audio into symbolic music reresentations, using no rior knowledge of the music to be recognized. This roblem remains comletely oen, esecially with olyhonic (several indeendent arts) music, where the state of the art remains rimitive. While there are many ways one can build reasonable data models quantifying how well a articular audio sniet matches a hyothesized collection of itches, what seems to be missing is the musical language model. If honemes and notes are regarded as the atoms of seech and music, there does not seem to be a musical equivalent of the word. Furthermore, while music follows simle logic and can be quite redictable, this logic is often cast in terms of higher-level constructs such as meter, harmony and motivic transformation. Comutationally tractable models such as note n-grams seem to contribute very little here, while a comutationally useful music language model remains uncharted territory. Our Listen module deals with the much simler situation in which the music score is known, giving the itches the soloist will lay along with their aroximate durations. Thus the score following roblem is one of alignment rather than recognition. Score following, otherwise known as on-line alignment, is more difficult than its off-line cousin, since an on-line algorithm cannot consider future audio data in estimating the times of audio events. Thus, one of the rincial challenges of on-line alignment is the trade-off between accuracy reorting the correct times of note events and latency the lag in time between the estimated note event time and the time the estimate is made. (Schwarz, 2003) gives a nice annotated bibliograhy of the many contributions to score following The Listen Model Our HMM aroach views the audio data as a sequence of frames, y 1, y 2,..., y T, with about 30 frames er second, while modeling these frames as the outut of a hidden Markov chain, x 1, x 2,..., x T. The state grah for the Markov chain, described in Figure 1, models the music as a sequence of sub-grahs, one for each solo note, which are arranged so that the rocess enters the start of the n + 1st note as it leaves the nth note. From the figure, one can see that each note begins with a short sequence of states meant to cature the attack ortion of the note. This is followed by another sequence of states with self-loos meant to cature the main body of the note, and to account for the variation in note duration we may observe, as follows. If we chain together m states which each either move forward, with robability, or remain in the current state, with robability q = 1, then the total number of state visits, L, (audio frames) sent in the sequence of m states has a negative binomial distribution P (L = l) = ( m 1 l 1 ) m q l m for l = m, m + 1,.... While convenient to reresent this distribution with a Markov chain, the asymmetric nature of the negative binomial is also musically reasonable: while it is common for an inter-onset interval (IOI) to be much longer than its nominal length, the reverse is much less common. For each note we choose the note arameters m and so that E(T ) = m/

4 and Var(T ) = mq/ 2 reflect our rior beliefs. If we have seen several erformances of the iece in question, we choose m and according to the method of moments so that the emirical mean and variance agree with the true mean and variance. Otherwise, we choose the mean according to what the score rescribes, while choosing the variance to cature our lack of knowledge. In reality, we use a wider variety of note models than deicted in the figure, with variants on the above model for short notes, notes ending with otional rests, notes that are rests, etc, though all following the same essential idea. The result is a network of thousands of states. Our data model is comosed of three features b t (y t ), e t (y t ), s t (y t ) assumed to be conditionally indeendent given the state P (b t, e t, s t x t ) = P (b t x t )P (e t x t )P (s t x t ). The first feature, b t, measures the local burstiness of the signal, articularly useful in distinguishing between note attacks and steady state behavior observe that we distinguished between the attack ortion of a note and and steady state ortion in Figure 1. The 2nd feature, e t measures the local energy, useful in distinguishing between rests and notes. By far, however, the vector-valued feature s t is the most imortant, as it is well-suited to making itch discriminations, as follows. We let f n denote the frequency associated with the nominal itch of the nth score note. As with any quasieriodic signal with frequency f n, we exect that the audio data from the nth note will have a magnitude sectrum comosed of eaks at integral multiles of f n. This is modeled by the Gaussian mixture model deicted in Figure 2 n (j) = H w h N(j; µ = hf n, σ 2 = (ρhf n ) 2 ) h=1 where h w h = 1 and N(j; µ, σ 2 ) is a discrete aroximation of a Gaussian distribution. We define s t to be the magnitude sectrum of y t, normalized to sum to constant value, C. If we believe the nth note is sounding in the tth frame, we regard s t as the histogram of a random samle of size C. Thus our data model becomes the multinomial distribution C! P (s t x t note n) = j s n (j) st(j) (1) t(j)! This model describes the art of the audio sectrum due to the soloist reasonably well. However, our actual signal will receive not only this solo contribution, j Figure 2. An idealized note sectrum modeled as a mixture of Gaussians. but audio generated by our accomaniment system as well. If the accomaniment audio contains frequency content that is confused with the solo audio, the result is the highly undesirable ossibility of the accomaniment system following itself in essence, chasing its own shadow. To a certain degree, the likelihood of this outcome can be diminished by turning off the score follower when the soloist is not laying; of course we do this. However, there is still significant otential for shadow-chasing since the itch content of the solo and accomaniment arts is often similar. Our solution to this difficulty is to directly model the accomaniment contribution to the audio signal we receive. Since we know what the orchestra is laying (our system generates this audio), we add this contribution to the data model. More exlicitly, if q t is the magnitude sectrum of the orchestra s contribution in frame t, we model the conditional distribution of s t using Eqn. 1, but with k,n = λ n + (1 λ)q t for 0 < λ < 1 instead of n. This addition creates significantly better results in many situations. The surrising difficulty in actually imlementing the aroach, however, is that there seems to be only weak agreement between the known audio that our system lays through the seakers and the accomaniment audio that comes back through the microhone. Still, with various averaging tricks in the estimation of q t, we can nearly eliminate the undesirable shadow-chasing behavior. freq

5 3.2. On-Line Interretation of Audio Perhas one of the main virtues of the HMM-based score follower is the grounding it gives to navigating the accuracy-latency trade-off. One of the worst things a score follower can do is reort events before they have occurred. In addition to the sheer imossibility of roducing accurate estimates in this case, the musical result often involves the accomanist arriving at a oint of coincidence before the soloist does. When the accomanist stes on the soloist in this manner, the soloist must struggle to regain control of the erformance, erhas feeling deserate and irrelevant in the rocess. Since the consequences of false ositives are so great, the score follower must be reasonably certain that a note event has already occurred before reorting its location. We handle this as follows. Every time we rocess a new frame of audio we recomute the forward robabilities, (x t y 1,..., y t ), for our current frame, t. Listen waits to detect note n until we are sufficiently confident that its onset is in the ast. That is, until P (x t start n y 1,..., y t ) τ In this exression, start n reresents the initial state of the nth note model, as indicated in Figure 1, which is either before, or after all other states in the model. Suose that t is the first frame where the above inequality holds. When this occurs, our knowledge of the note onset time can be summarized by the function of t: P (x t = start n y 1,..., y t ) which we comute using the forward-backward algorithm. Occasionally this distribution conveys uncertainty about the onset time of the note, say, for instance, if it has high variance or is bimodal. In such a case we simly do not reort the onset time of the articular note, believing it is better to remain silent than rovide bad information. Otherwise, we estimate the onset as ˆt n = arg max t t P (x t = start n y 1,..., y t ) (2) and deliver this information to the Predict module. Several videos demonstrating the ability of our score following can be seen at the aforementioned web site. One of these simly lays the audio while highlighting the locations of note onset detections at the times they are made, thus demonstrating detection latency what one sees lags slightly behind what one hears. A second video shows a rather eccentric erformer who ornaments wildly, makes extreme temo changes, lays wrong notes, and even reeats a measure, thus demonstrating the robustness of the the score follower. 4. Predict: Modeling Musical Timing As discussed in Section 2, we believe a urely resonsive accomaniment system can not achieve accetable coordination of arts in the range of common ractice classical music we treat, thus we choose to schedule our accomaniment through rediction rather than resonse. Our aroach is based on a model for musical timing. In develoing this model, we begin with three imortant traits we believe such a model must have. 1. Since our accomaniment must be constructed in real time, the comutational demand of our model must be feasible in real time. 2. Our system must imrove with rehearsal. Thus our model must be able to automatically train its arameters to embody the timing nuances demonstrated by the live layer in ast examles. This way our system can better anticiate the future musical evolution of the current erformance. 3. If our rehearsals are to be successful in guiding the system toward the desired musical end, the system must sightread (erform without rehearsal) reasonably well. Otherwise, the layer will become distracted by the oor ensemble and not be able to demonstrate what she wants to hear. Thus there must be a neutral setting of arameters that allows the system to erform reasonably well out of the box The Timing Model We first consider a timing model for a single musical art. Our model is exressed in terms of two hidden sequences, {t n } and {s n } where t n is the time, in seconds, of nth note onset and s n is the temo, in seconds er beat, for the nth note. These sequences evolve according to s n+1 = s n + σ n (3) t n+1 = t n + l n s n + τ n (4) where l n is the length of the nth event, in beats. With the udate variables, {σ n } and {τ n }, set to 0, this model gives a literal and robotic musical erformance with each inter-onset-interval, t n+1 t n, consuming an amount of time roortional to its length in beats, l n. The introduction of the udate variables allow time-varying temo through the {σ n }, and elongation or comression of note lengths with the {τ n }. We further assume that the {(σ n, τ n ) t } are indeendent with (σ n, τ n ) t N(µ n, Γ n ) and (s 1, t 1 ) t N(µ 0, Γ 0 ), thus leading to a joint Gaussian model on all model

6 3 3 Solo Accom Listen Udates Comosite Accom Figure 3. To: Two musical arts generate a comosite rhythm when suerimosed. Bot: The resulting grahical model arising from the comosite rhythm. variables. The rhythmic interretation embodied by the model is exressed in terms of the {µ n, Γ n } arameters. In this regard, the {µ n } vectors reresent the tendencies of the erformance where the layer tends to seed u (σ n < 0), slow down (σ n > 0), and stretch (τ n > 0), while the {Γ n } matrices cature the reeatability of these tendencies. It is simlest to think of Eqns. 3-4 as a timing model for single musical art. However, it just as reasonable to view these Eqns. as a timing model for the comosite rhythm of the solo and orchestra. That is, consider the situation, deicted in Figure 3, in which the solo, orchestra, and comosite rhythms have the following musical times (in beats): solo 0 1/3 2/3 1 4/3 5/3 2 accom 0 1/2 1 3/2 2 com. 0 1/3 1/2 2/3 2 4/3 3/2 5/3 2 The {l n } for the comosite would be found by simly taking the differences of rational numbers forming the comosite rhythm: l 1 = 1/3, l 2 = 1/6, etc. In what follows we regard Eqns. 3-4 as a model for this comosite rhythm. The observable variables in this model are the solo note onset estimates roduced by Listen and the known note onsets of the orchestra (our system constructs these during the erformance). Suose that n indexes the events in the comosite rhythm having associated solo notes, estimated by the {ˆt n }. Additionally, suose that n indexes the events having associated orchestra notes with onset times, {o n }. We model ˆt n = t n + ɛ n Figure 4. A sectrogram of the oening of the 1st movement of the Dvoˆrák Cello concerto. The horizontal axis of the figure reresents time while the vertical axis reresents frequency. The vertical lines show the note times for the orchestra. o n = t n + δ n where ɛ n N(0, ρ 2 s) and δ n N(0, ρ 2 o). The result is the Gaussian grahical model deicted in the bottom anel of Figure 3. In this figure the row labeled Comosite corresonds to the {(s n, t n )} variables of Eqns. 3-4, while the row labeled Udates corresonds to the {(σ n, τ n )} variables. The Listen row is the collection of estimated solo note onset times, {ˆt n } while the Accomaniment row corresonds to the orchestra times, {o n } The Model in Action With the model in lace we are ready for real-time accomaniment. In our first rehearsal we initialize the model so that µ n = 0 for all n. This assumtion does not reclude the ossibility of the model correctly interreting and following temo changes or other rhythmic nuances of the soloist; rather, it states that we exect the timing to evolve strictly according to the temo when looking forward. In real-time accomaniment, our system is concerned only with scheduling the currently ending orchestra note time, o n. The time of this note is initially scheduled when we lay the revious orchestra note, o n 1. At this oint we comute the new mean of o n, conditioning o n 1 and whatever other variables have been observed, and schedule o n accordingly. While we wait for the currently-scheduled time to occur, the Listen module may detect various solo events, ˆt n. When this haens we recomute the mean of o n, conditioning on this new information. Sooner or later the actual clock time will catch u to the currently-scheduled time, at which oint the orchestra note is layed. Thus an or-

7 chestra note may be rescheduled many times before it is actually layed. A articularly instructive examle involves a run of many solo note culminating in a oint of coincidence with the orchestra. As each solo note is detected we refine our estimate of the desired oint of coincidence, thus gradually honing in on the this oint of arrival. It is worth noting that very little harm is done when Listen fails to detect a solo note. We simly redict the ending orchestra note conditioning on the variables we have observed. The web age given before contains a video that demonstrates this rocess. The video shows the estimated solo times from our score follower aearing as green marks on a sectrogram. Predictions of our accomaniment system are shown as analogous red marks. One can see the ending accomaniment event jiggling as new solo notes are estimated, until finally the currently-redicted time asses. The role of Predict is to schedule accomaniment notes, but what does this really mean in ractice? Recall that our rogram lays audio by hase-vocoding (time-stretching) an orchestra-only recording. A timefrequency reresentation of such an audio file for the 1st movement of the Dvoˆrák Cello concerto is shown in Figure 4. In rearing this audio for our accomaniment system, we erform an off-line score alignment to determine where the various orchestra notes occur, as marked with vertical lines in the figure. Scheduling a note simly means that we change the hase-vocoder s lay rate so that it arrives at the aroriate audio file osition (vertical line) at the scheduled time. Thus the lay rate is continually modified as the erformance evolves. After one or more rehearsals, we adat our timing model to the soloist to better anticiate future erformances. To do this we first erform an off-line estimate of the solo note times using Eqn. 2, only conditioning on the entire sequence of frames, y 1,..., y T, using the forward-backward algorithm. Using one or more such rehearsals we can iteratively reestimate the model arameters {µ n } using the EM algorithm, resulting in both measurable and erceivable imrovment of rediction accuracy. While, in rincile, we can also estimate the {Γ n } arameters, we have observed little or no benefit from doing so. In ractice we have found the soloist s interretation to be a bit of a moving target. At first this is because the soloist tends to comromise somewhat in the initial rehearsals, ulling the orchestra in the desired direction, while not actually reaching the target interretation. But even after the soloist seems to settle down to a articular interretation on a given day, we often observe further drift of the interretation over subsequent meetings, due to the changeable nature of musical ideas. For this reason we train the model using the most recent several rehearsals, thus facilitating the continually evolving nature of musical interretation. 5. Musical Exression and Machine Learning Our system learns its musicality through osmosis. If the soloist lays in a musical way and the orchestra manages to closely follow the soloist, then we hoe the orchestra will inherit this musicality. This manner of learning by imitation works well in the concerto setting, as the division of authority between the layers is rather extreme, mostly granting the right of way to to the soloist. However, many other musical scenarios do not so clearly divide the roles into leader and follower. Our system is least at home in this middle ground. Our timing model of Eqns. 3-4 is over-arametrized, with more degrees of freedom than there are notes. We make this modeling choice because it is hard know exactly which degrees of freedom are needed ahead of time, so we use the training data from the soloist to hel sort this out. Unnecessary learned arameters may contribute some noise to the resulting timing model, but the overall result is accetable. While this aroach works reasonably well in the concerto setting, it is less reasonable when our system needs a sense of musicality that acts indeendently, or erhas even in oosition, to what other layers do. Such a situation occurs with the early-stage accomaniment roblem discussed in Section 1, as here one cannot learn the desired musicality from the live layer. Perhas the accomaniment antithesis of the concerto setting is the oera orchestra, in which the accomanying ensemble is often on equal footing with the soloists. We observed the nadir of our system s erformance in an oera rehearsal where our system served as rehearsal ianist. What these two situations have in common is that they require an accomanist with indeendent musical knowledge and goals. One ossible line of imrovement is simly decreasing the model s freedom surely the layer does not wish to change the temo and aly temo-indeendent note length variation on every note. One ossibility adds a hidden discrete rocess that chooses, for each note, between three ossibilities: no variation of either kind, or variation of either temo or note length. Of these, the choice of neither variation would be the most likely, thus biasing the model toward simler musical

8 interretations, adoting the oint of view of Ockham s razor with regard to interretation. While exact inference is no longer ossible with such a model, we exect that one can make aroximations that will be good enough to realize the full otential of the model, whatever that is. Perhas a more interesting aroach analyzes the musical score to choose the locations requiring additional degrees of freedom. One can think of this aroach as adding joints to the musical structure so that it deforms into musically-reasonable shaes as a musician alies external force. Here there is an interesting connection with the work on exressive synthesis, such as (Widmer & Goebl, 2004), in which one algorithmically constructs an exressive rendition of a reviously-unseen iece of music, using ideas of machine learning. One idea here is to associate various score situations, defined in terms of local configurations of measurable score features, with interretive actions. The associated interretive actions are learned by estimating timing and loudness arameters from a erformance corus, over all equivalent score locations. Such aroaches are far more ambitious than our resent aroach to musicality, as they try to understand exression in general, rather than in a secific musical context. The understanding and synthesis of musical exression is surely one of the grand challenges facing music-science researchers, and while rogress has been achieved in recent years, it would still be fair to call the roblem oen. One of the rincial challenges here is that one cannot directly ma observable surfacelevel attributes of the music, such as itch contour or local rhythm context, into interretive actions, such as delay, or temo or loudness change. Rather, there is a murky intermediate level in which the musician comes to some understanding of the musical meaning, and on which the interretive decisions are conditioned. This meaning comes from different asects of the music. Some comes from structural asects, for examle, as the way the end of a section or hrase is reflected with a sense of closure. Some meaning comes from rosodic asects, analogous to seech, such as anticiation and oints of arrival. A third asect of meaning describes an overall character or affect of a section of music, such as excited or calm. While there is no official taxonomy of musical interretation, most discussions on this subject stem from such intermediate identifications and the interretive actions they require. From the machine learning oint of view it is imossible to learn anything useful from a single examle, thus one must grou together many examles of the same musical situation in order to learn their associated interretive actions. Thus it seems natural to model the music in terms of some latent variables that imlicitly categorize individual notes or sections of music. What should the latent variables be, and how can one describe the deendency structure among them? While we cannot answer these questions, we see in them a good deal of deth and challenge, and recommend this roblem to the musically inclined members of the ML community with great enthusiasm. References Cont, A.,, Schwarz, D., and Schnell, N. From boulez to ballads: training ircam s score follower. In Proc. of the International Comuter Music Conference, , Dannenberg, R. and Mukaino, H. New techniques for enhanced quality of comuter accomaniment. In Proc. of the 1988 International Comuter Music Conference, , Dannenberg, Roger and Mont-Reynaud, Bernard. Following an imrovisation in real time. In Proc. of the 1987 International Comuter Music Conference, , Flanagan, J. L. and Golden, R. M. Phase vocoder. Bell Systems Technical Journal, 45(Nov): , Franklin, Judy. Imrovisation and learning. In Advances in Neural Information Processing Systems 14. MIT Press, Cambridge, MA, Lie, C. Real-time interaction among comosers, erformers, and comuter systems. Information Processing Society of Jaan, SIG Notes, 123:1 6, Pachet, Francois. Beyond the cybernetic jam fantasy: The continuator. IEEE Comuter Grahics and Alications, 24(1):31 35, Rahael, C. Current directions with music lus one. In Proc. of the 6th Sound and Music Comuting Conference, , Rowe, Robert. Interactive Music Systems. MIT Press, Schwarz, D. Score following commented bibliograhy, URL htt:// tems-reel/suivi/bibliograhy.html. Widmer, G. and Goebl, W. Comutational models for exressive music erformance: The state of the art. Journal of New Music Research, 2004.