YQX Plays Chopin By G. Widmer, S. Flossmann and M. Grachten AssociaAon for the Advancement of ArAficual Intelligence, 2009 Presented by MarAn Weiss Hansen QMUL, ELEM021 12 March 2012 Contents IntroducAon Expressive Music Performance How YQX Works Results QMUL ELEM021 PresentaAon: M. W. Hansen 1
IntroducAon AI research related to arasac behavior Computer program plays with expression: YQX Goal: InvesAgate and understand music performance Results: Rencon compeaaon 2008 Machines can give interesang insights even if they aren t creaave themselves Expressive Music Performance Shaping a musical piece by varying important parameters like tempo and dynamics Humans do not play music mechanically, they interpret the printed score Most important parameters: Timing, tempo changes, dynamics and araculaaon ArAsAc freedom is not unlimited Tension between creaave freedom and various constraints QMUL ELEM021 PresentaAon: M. W. Hansen 2
How YQX Works Bayesian model describes relaaonship between score context (input) and expressive parameters (output) Task: Learn to predict three expressive features (Aming, dynamics, araculaaon) Training data: Human piano performances 13 Mozart pieces performed by R. BaAk 39 Chopin pieces performed by N. Magaloff How YQX Works a Features IR-Label: Process IR-Closure: -0.4332 Pitch-Interval: Rhythmic context: s-s Duration Ratio: 0.301 b score discrete features Q continuous features X Targets Timing: 0.036 Articulation: 1.22 Loudness: 0.2 performance continuous target Y QMUL ELEM021 PresentaAon: M. W. Hansen 3
How YQX Works For each melody note properaes are calculated ImplicaAon- realizaaon analysis à IR- label IR- closure Targets are computed from the training data Timing ArAculaAon Loudness Results Rencon 2008 contest Videos: www.cp.jku.at/projects/yqx A computer program is to play two never seen before piano pieces in an expressive way Chopin- like piece and Mozart- like piece Scores contain notes and expression marks, but no informaaon about structure, chords etc. Output MIDI files which are played back on a computer- controlled piano No hand- ediang allowed Performance is graded by a human audience and the composer of the pieces QMUL ELEM021 PresentaAon: M. W. Hansen 4
160 Results YQX tempo curve for MyNocturne (soprano voice) 140 a Tempo (BPM) 120 100 80 60 1 5 9 13 17 21 25 29 33 Score time (bars) b Loudness (MIDI velocity) 90 [a] 80 70 YQX dynamics curve for MyNocturne (soprano voice) [a ] [b ] 60 50 40 [b] 30 1 5 9 13 17 21 25 29 33 Score time (bars) Articles 1.5 1 Figure 4. My Nocturne as Performed by YQX. A. Tempo. B. Dynamics. The horizontal axes represent the logical score time, that is, the position in the piece, counted in terms of bars from the beginning. In the tempo plot, the vertical axis gives the local tempo YQX played at that point in time (in terms of beats [eighth notes] per minute). In the dynamics plot, the vertical axis gives the loudness of the current melody note (in terms of MIDI velocity). Only notes of the melody of the piece (the highest part in the right hand) are indicated. See the text for an explanation of the specially marked passages. Results starting with bar 11 (see figure 8 for YQX s renderings). Here, we see a speedup over the first part of sage [a]. Unfortunately, there is no corresponding First-order phase plane, bars 3-5 softening at the end [b ]: First-order YQX chose phase plane, to play bars 9.5 the - 11 closing B 1.5 in bar 6 louder and, worse, with a staccato, as can be clearly heard in the video a rather clear musical 1 mistake. the phrase (the agitated gesture consisting of six sixteenth notes in phrase B1, and the corresponding septuplet in B2), followed by a slowing down over the course of the trill (tr), followed by a speedup towards the beginning of the next phrase. The essential difference here is that, in repeat 2, YQX introduces a strong (and very natural-sounding) ritardando towards the end, because of the approaching end of the first section of the piece. To make this slowing down towards the end appear The tempo strategy YQX chose for the entire first section 0.5(from bar 1 to the beginning of the 0.5 middle section in bar 13) deserves to be analyzed in a bit more detail. To facilitate an intuitive visual understanding of the high-level trends, we will 0 0 present this analysis in the phase plane (see figures 7 through -0.5 9). Phase-plane plots, known from -0.5 40 even more dramatic, 6 the preceding speedup is dynamical systems theory, display the behavior of AI MAGAZINE proportionately stronger than in the previous a system by plotting variables describing the state phrase. Overall, we again see YQX treating similar of the system against each other, as opposed to phrases in analogous ways, with the second rendition plotting each variable against time. It is common being less pronounced than the first, and with.5 in phase-plane -0.5 plots that one 0axis represents 0.5 the.5 1 0.2 0.4 0.6 0.8 1 1.2 1.4 some variation to account for a different musical derivative of the other axis. For example, dynamical -0.2 0 context. This can be seen even more clearly in figure systems that describe physical motion of objects First-order 9, which phase shows plane, the bars 5 entire - 7 passage (bars 9.5 are typically visualized First-order by plotting phase velocity plane, against bars 11-12.5 12.5) in one plot. One might be tempted to say 1.5 displacement. In the context of expressive performance, 1.5 that YQX is seeking to produce a sense of unity in phase-plane representations have been variation. proposed for 1 the analysis of expressive timing 1 Of course, YQX s performance of the complete (Grachten et al. 2008), because they explicitly visualize piece is far from musically satisfying. Listening to the dynamic aspects of performance. Expres- 0.5 0.5 the entire performance, we hear passages that sive gestures, manifested as patterns of variation make good musical sense and may even sound natural in timing, tend to be more clearly revealed in this or humanlike, but we also observe quite a 0 way. The sidebar Visualizing Expressive Timing in 0 number of blatant mistakes or unmusical behaviors, the Phase Plane gives some basic information things that no human pianist would ever about the computation -0.5 and meaning of smoothed -0.5 think of doing. 7 But overall, the quality is quite timing trajectories in the phase plane. A more surprising, given that all of this was learned by the detailed description of the computation process machine. and the underlying assumptions can be found in Grachten et al. (2009). We will also encounter.5 phase-plane.5plots later in this article, when we -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4-0.5 0 0.5 1 Is YQX Creative? compare the timing behavior of famous pianists. Figure 7 shows how YQX chose to play two very similar passages at the beginning of the piece: the first appearance of the main theme in bars 3 4, and Surely, expressive performance is a creative act. Beyond mere technical virtuosity, it is the ingenuity with Figure 7. YQX s Expressive Timing which (Tempo great artists Changes) portray in the a Phase piece Plane. in a new Figure 8. YQX s Expressive Timing (Tempo Changes) over Bars 9.5 11 way and illuminate aspects we have never noticed the immediately following variation of the theme (Left) and 11 12.5 (Right), Respectively. Left: Bars 3 4 of My Nocturne. before that Right: commands Bars 5 6. Horizontal our admiration. axis: (smoothed) While (bars 5 6); the two phrases are marked as A1 and tempo in beats per minute YQX s (bpm). performances Vertical axis: First may derivative not exactly of tempo command (that A2 in the score in figure 3. Overall, we see the same is, amount of accelerando or ritardando). Empty circles indicate the beginnings our admiration, they are certainly not unmusical general tempo pattern in both cases: a big arc a of new bars; black dots mark beat times (each of the 6 eighth note beats within a bar); the end of a speedup ful, planned followed behavior by a slow-down that even results (accelerandoritardando) thagant artifact, is solely characterizes an epiphenomenon the first part (bar), in an ele- (for curve the is most marked part), by a they diamond seem (to to indicate betray some the direc- musical understanding, and they are sometimes sur- emerg- tion). The tempo and acceleration values have been normalized to give a µ = QMUL ELEM021 PresentaAon: M. W. Hansen 5 0.0 and a σ = 1.0 over the entire piece; the absolute values thus do not have a followed ing from by low-level, prising. Can YQX be said to be creative, then? a further ritardando local decisions decisions with an interme-on direct musical interpretation. diate what little tempo local and speedup loudness (the to apply loop in at a the given phase point In our view (and certainly in the view of the general public), creativity has something to do with plot; in note the piece, that the without final segment much of regard the curve to the (leading musical to the context. final diamond) already pertains to the larger intentionality, with a conscious awareness of form, sixteenth-note figure in bar 3, which YQX accompanies with a crescendo-decrescendo in the dynamics structure, aesthetics, with imagination, with skill, first beat With of YQX, the next we have bar). Viewing no idea the why two the corresponding chose to passages do what side it did. by Of side, course, we clearly it would see be that pos- system and with the ability of self-evaluation (see also domain Boden (see 1998, indications Colton 2008, [a] in Wiggins figure 2006). 4). Note The also previous they sible were in principle played with to trace analogous back the shapes, reasons but for in these a the extreme analysis softening of the Chopin of the performance last note of seems this to less decisions, pronounced but they or dramatic would not way be the very second revealing time phrase suggest (the that last YQX note has of bar a sense 4), which of unity gives and a very coherence, variation, around: they would the second essentially shape is point a kind to of the smaller existence replica musical of the first. situations in the training corpus that are of natural ending to and the so phrase on. The [b]. fact Both is, of however, these choices that YQX are musically has no notion very sensible. whatsoever Note also of abstract how somehow A similar picture similar emerges to the musical when we passage analyze currently the the following passage (bar 5 see [a ]) is played next pair of corresponding (though again not iden- Articles
EvaluaAon The authors spend a lot of space talking about whether YQX is creaave or not YQX has a sense of unity and coherence, variaaance etc. But no noaon of abstract musical concepts like structure and form, it only knows about low- level, local decisions Which is of course because the features chosen were based on low- level informaaon The authors: CreaAve behavior may emerge as a consequence of high complexity EvaluaAon The true goal: Studying creaave human behavior with the help of intelligent computer methods Beyond YQX: The agenda is to find commonaliaes and differences between performances of different arasts by performing focused computaaonal invesagaaons Performances are analysed using machine learning (YQX) EvaluAons show that performance rules are surprisingly predicave (personal style is stable) Modelling expression at mulaple levels of the musical phrase hierarchy leads to beger predicaons QMUL ELEM021 PresentaAon: M. W. Hansen 6
EvaluaAon Future: Extend YQX step by step QuanAfy the relevance of various components and layers of the model Two specific direcaons are pursued to improve YQX 1: Extend YQX with local temporal contexts 2: MulAple structural levels Use N. Magaloff s performances of Chopin s works to build a computaaonal model of an accomplished arast RepresenAng expressive Aming in the phase plane is a new idea which needs further invesagaaon QuesAons? QMUL ELEM021 PresentaAon: M. W. Hansen 7