COMPUTATIONAL INVESTIGATIONS INTO BETWEEN-HAND SYNCHRONIZATION IN PIANO PLAYING: MAGALOFF S COMPLETE CHOPIN

COMPUTATIONAL INVESTIGATIONS INTO BETWEEN-HAND SYNCHRONIZATION IN PIANO PLAYING: MAGALOFF S COMPLETE CHOPIN Werner Goebl, Sebastian Flossmann, and Gerhard Widmer Department of Computational Perception Johannes Kepler University Linz, Austria ABSTRACT The paper reports on first steps towards automated computational analysis of a unique and unprecedented corpus of symbolic performance data. In particular, we focus on between-hand asynchronies an expressive device that plays an important role particularly in Romantic music, but has not been analyzed quantitatively in any substantial way. The historic data were derived from performances by the renowned pianist Nikita Magaloff, who played the complete work of Chopin live on stage, on a computer-controlled grand piano. The mere size of this corpus (over 32, performed notes or almost 1 hours of continuous performance) challenges existing analysis approaches. The computational steps include score extraction, score-performance matching, definition and measurement of the analyzed features, and a computational visualization tool. We then present preliminary data to demonstrate the potential of our approach for future computational modeling and its application in computational musicology. 1 INTRODUCTION This paper presents research towards automated computational analysis of large corpora of music performance data. In particular, we give a first report on a computational approach to analyzing a unique corpus of historic performance data: basically the complete work of Chopin, performed by the renowned pianist Nikita Magaloff. Corpora of that size hundreds of thousands of played notes require a level of automation of analysis that has not been accomplished so far. We describe the required processing steps, from converting scanned scores into symbolic notation, to score-performance matching, definition and automatic measurement of between-hand asynchronies, and a computational visualization tool for exploring and understanding the extracted information. SMC 29, July 23 25, Porto, Portugal Copyrights remain with the authors As the two hands of a pianist have the possibility to produce different musical parts independently, the between-hand asynchronies yield a spectrum of artistic expression ranging from the fairly constrained melody lead effect [1] to bass anticipations [5], or the earlier type of tempo rubato [3]. Towards the end of the paper, we present preliminary results on the between-hand asynchronies in Magaloff s complete Chopin to demonstrate the scope of insights that such large corpora can offer. Finally, we discuss the future pathways of this research endeavor and its potential for computational modeling and musicological investigation. 2 THE CHOPIN CORPUS The analyzed Chopin corpus comprises live concert performances by the Georgian-Russian pianist Nikita Magaloff (1912 1992), who played almost the entire solo repertoire of Chopin in a series of 6 recitals between January and May 1989 at the Mozart-Saal of the Wiener Konzerthaus 1 in Vienna on a Bösendorfer computer-controlled grand piano. In this unprecedented project, Magaloff, by that time already 77 years old, performed all works of Chopin for solo piano that appeared in print during Chopin s life time, keeping a strict ascending order by opus number, starting with the Rondo, Op. 1 up to the three Waltzes Op. 64, including the three Sonatas, 41 Mazurkas, 25 Préludes, 24 Études, 18 Nocturnes, 8 Waltzes, 6 Polonaises, 4 Scherzos, 4 Ballades, 3 Impromptus, 3 Rondos, and other works (Variations brillantes, Bolero, Tarantelle, Allegro de Concert, Fantaisie, Berceuse, Barcarole, and Polonaise-Fantaisie). 2 Magaloff played these recitals on a Bösendorfer SE computer-controlled grand piano that recorded his performances onto computer hard disk. The SE format stores the performance information in a symbolic format with high precision, providing detailed information on the onset and off- 1 This concert hall provides about 7 seats, http://www. konzerthaus.at. 2 The works not played were either piano works with orchestra accompaniment (Op. 2, 11, 13, 14, 21, and 22), works with other instruments (Op. 3, 8 and 65), or works with higher (op. posth., starting from Op. 66, the Fantaisie-Impromptu) or no opus numbers. Page 291

set timing of each played tone, the dynamics, and the pedalling. The entire corpus comprises more than 15 individual pieces, over 323, performed notes or almost 1 hours of continuous performance. 3 COMPUTATIONAL ANALYSIS OF PERFORMANCE DATA 3.1 Score Extraction In order to analyze symbolic performance data automatically, the performances have to be connected to the corresponding musical scores (score-performance matching). As symbolic scores were not available for the complete work of Chopin, the first step was to extract this information from the printed music scores. We used a music recognition software ( SharpEye ) to convert the 946 pages of scanned music into a musicxml representation. Extensive manual verification of the conversion process was necessary to eliminate a large number of conversion errors, as well as postcorrection of conversion incapabilities of the used software (e.g., ottava lines, parts crossing staffs, etc.). 3.2 Score Performance Matching The symbolic scores were then matched to Magaloff s performances employing a matching algorithm based on an edit distance metric [4]. Also the matching results had to be inspected and corrected manually with an interactive graphical user interface that displays the note-by-note match between the score information and the performance. All uncorrectly played notes or performed variants were identified and labeled. (This, by the way, will also make it possible to perform large-scale, in-depth analyses of the kinds of errors accomplished pianists make.) 3.3 Defining and Measuring Asynchronies Our aim was to analyze the between-hand asynchronies of notes that are notated as nominally simultaneous in the score (that is, all tones belonging to the same score event ). To that end, we first needed to compute these asynchronies automatically from the corpus. The staff information of the musical notation (upper versus lower staff) was used to calculate the between-hand asynchronies. As the performance data does not contain information about what hand played what parts of the music, we assumed that overall the right hand played the upper staff tones and the left hand the lower. 3 We define a between-hand asynchrony as follows: For all notes belonging to the same score event, subtract the on- 3 Certainly, there are numerous passages where this simple assumption is wrong or not likely to be true, but given the sheer size of the data set, the potential bias may be tolerable. set timing of each upper-staff note from the onset timing of each lower-staff note ( lower minus upper ). Thus, positive asynchrony values indicate that the upper staff (right hand) is early, while negative numbers denote bass anticipations (left hand early). Multiple asynchronies within one score event were averaged for subsequent data analyses. All notated arpeggios, ornaments, trills, or grace notes were excluded from our preliminary data analysis (about 1% of the entire data), as these cases feature special and usually larger asynchronies than regular score events. These special cases deserve a separate detailed analysis that would exceed the scope of the present paper. 3.4 Computational Visualization For a first intuitive analysis and understanding of this huge amount of measurement data, adequate visualization methods are needed. Thus, we developed a dedicated computational visualization tool. A screenshot is presented in Figure 1. It comprises three panels on top of each other, sharing the same time axis. The upper panel shows the individual tempo curves of the two hands; the middle panel shows the average asynchronies for each score event that contained simultaneous notes in each staff; and the lower panel features a piano-roll representation of the performances with the relevant notes connected by vertical lines. The color of these lines is either red (indicating a right-hand lead) or green (indicating a left-hand lead). The grey area in the middle panel marks a range of +/ 3 ms within which asynchronies are not likely to be perceived as such [2]. Furthermore, the tool indicates occurrences of bass anticipations ( B.A., lower panel) and out-of-sync regions (horizontal bars, middle panel; for description see below). 4 PRELIMINARY RESULTS In the following, we give a first glimpse of the scope of results that such large-scale analyses may yield. 4.1 Overall Asynchronies The distribution of all asynchronies between the two hands is shown in Figure 2, including the mean and the mode value. The positive mode value reflects an overall tendency for the right hand to be early, which is most likely attributable to the well-known melody lead effect [1]. Moreover, the mean value is slightly below the mode value reflecting a slightly skewed histogram towards the left side. Particularly in the region of 1 to 3 ms there is a slight increase of values, most likely due to frequent bass anticipations (red line, see below). The asynchrony distributions of the individual pieces vary considerably and may depend on the specifics of the pieces. The pieces played most synchronously by Magaloff are those Page 292

TEMPO (bpm) 24 16 12 8 6 4 3 Op. 27 No. 2 played by Nikita Magaloff 522 524 526 528 53 532 534 536 538 Right hand Left hand ASYNCH (ms) 2 1 1 2 11 522 524 526 528 53 532 534 536 538 ) 1 9 8 PITCH 7 6 265. 226. 224. 255. 264. 157. 228. B.A. 5 4 B.A. B.A. 3 2 522 524 526 528 53 532 534 536 538 TIME (s) Figure 1. Screenshot of a computational visualization tool showing bars 5 54 of Chopin s Nocturne Op. 27 No. 2 performed by Magaloff. The tempo curves of the two hands are plotted in the upper panel, the asynchronies in middle (positive values indicate an early right hand; negative an early left hand; the grey area sketches the +/ 3-ms region around zero), and a piano roll representation is shown in the lower panel. All tones of computed asynchronies are connected by (almost) vertical lines that are plotted in red when the melody (right hand) was ahead, green when it lagged. Event wise Count 12 1 8 6 4 2 Mean=4.388 Mode=13. 5 25 25 5 Between hand Asynchronies (ms) Figure 2. Histogram of the signed between-hand asynchronies per event over the entire Chopin corpus (displaying a total of 163,28 asynchronies). The distribution of bass anticipations (see Section 4.2) is drawn by a red line on the left part of the histogram. that feature predominantly chordal textures (Op. 4-1, 28-9, 28-2, 1-2, see Figure 3); the least synchronous pieces are those that have strong melodic textures that leave room for artistic interpretation. There is a significant tempo effect within the investigated pieces. Figure 3 shows the mean absolute (unsigned) asynchronies per piece (a) and the standard error of the asynchronies (b) against the average event rate (in events per second). An event rate value was computed for each score event by counting the performed events (chords) within a time window of 3 seconds around it. The average event rate is the piece-wise mean of those values. We found that the faster the piece, the lower is the absolute asynchrony and also the lower is the variability of the asynchronies, which suggests that Magaloff takes more room to employ expressive asynchronies in slower pieces than faster pieces. 4.2 Bass Anticipations A bass anticipation is labeled as such when the lowest tone of a lower-staff score event is more than 5 ms ahead of the mean onsets of the upper-staff tones of that event. The Page 293

a) b) mean Unsigned Asynchrony (ms) SE of Asnchronies (ms) 8 7 6 5 4 3 2 1 28 2 1 6 45 62 1 48 2 58 3 28 13 27 2 55 2 15 2 9 1 27 1 9 3 37 1 17 4 63 2 r=.28*** n=15 p<.1 28 4 9 232 124 1 1 12 25 2 55 1 4 3 5 2 57 1 9 28 24 32 2 62 2 28 7 15 1 25 1 7 2 3 3 34 2 51 35 1 54 17 2 25 7 59 2 6 59 1 56 3 64 2 58 1 1 1 58 2 63 3 33 3 3 148 1 6 2 17 3 24 4 24 3 7 5 1 3 6 4 2 2319 29 28 19 28 6 47 3652 31 25 11 3 4 26 15 3 59 3 38 28 21 5 28 23 34 1 25 3 35 4 35 3 41 1 28 15 28 17 33 4 4 261 5 1 37 2 6 1 1 7 341 3 56 2 64 3 7 1 4 1 12 25 8 43 58 4 28 3 33 1 15 3 17 1 28 1 24 235 25 1 49 28 5 16 4 2 28 11 39 42 64 1 56 1 46 28 16 63 1 41 4 44 53 34 3 1 7 25 6 1 4 1 5 7 4 28 14 4 4 1 8 3 2 41 2 18 28 22 33 2 6 3 26 2 25 4 28 2 25 9 28 12 28 18 1 2 28 9 4 1 2 4 6 8 1 12 2 15 1 5 28 2 r=.417*** n=15 p<.1 28 7 1 6 45 63 2 28 1 33 3 37 1 28 13 28 4 15 2 28 6 58 3 7 2 6 4 62 17 4 28 23 9 1 9 217 2 48 2 3 1 28 24 24 1 32 1 4 3 27 1 3 3 55 2 5 2 62 2 27 2 24 3 28 21 7 5 55 1 56 2 25 7 9 3 28 11 1 12 33 1 32 2 17 1 15 1 9 28 3 17 3 6 1 1 3 4 2 6 2 7 3 24 4 7 4 15 3 28 9 28 15 41 1 56 3 63 3 59 2 58 2 3 4 34 2 59 1 35 3 5 1 41 3 28 2 26 1 37 2 33 4 48 1 35 2 3635 1 28 22 24 2 25 419 1 7 6 3 7 1 25 1 4 112 223 25 8 28 12 28 5 1525 9 28 14 25 3 1 1 25 1 1 2 25 2 1 525 11 28 173 2 5 3 41 4 4 2 51 57 63 1 64 2 59 3 64 1 6 56 1 47 38 34 1 34 3 1 4 1 8 28 16 28 18 41 2 58 1 61 64 3 54 29 1833 2 31 16 42 25 6 28 19 35 4 4 1 26 2 44 53494652 39 4 4 43 58 4 2 4 6 8 1 12 Figure 3. Absolute (unsigned) asynchronies (a) and standard error (SE) of the mean asynchronies (b) against the mean event rate per piece. The numbers refer to the opus numbering of the pieces. overall distribution of the bass leads is shown in Figure 2 (red line) on the left side of the histogram, and the invididual pieces are shown in Figure 4. The proportion of bass anticipations is lowest on average for the Etudes, the Preludes, and the Rondos (well below an average of 1% of the events), and highest in the Mazurkas and the Nocturnes (almost 2%). No bass anticipations were found particularly in the Preludes (16 out of 25 did not contain bass anticipations) and the Etudes (7 of 24). An exception is the Prelude Op. 28 No. 2, which exhibits both the highest mean asynchronies and the largest proportion of bass anticipations among all pieces (clearly visible in Figures 3 and 4). This very slow and short piece features a constant 1/8-note accompaniment with a single-note melody above it. The sad character and the slow tempo may be the reason for the high temporal independence of the melody in Proportion of Bass Anticipations.2.15.1.5 28 2 63 2 1 6 17 4 r=.349*** n=115 p<.1 9 2 4 2 58 2 37 1 28 13 41 124 1 62 1 15 2 58 3 59 2 27 2 17 3 33 4 6 28 23 7 2 26 1 17 2 15 1 17 124 4 28 21 3 3 5 248 1 57 56 3 25 1 4 1 38 15 3 4 3 34 2 36 4 2 6 1 7 5 5 1 2 9 1 9 3 2 23 25 8 48 2 35 1 25 7 24 2 28 11 25 4 18 28 22 31 3 1 1 3 6 26 3 12 19 28 129 1 16 4 4 1 12 1 4 1 5 25 11 32 1 28 17 32 2 41 4 4447 46 5 3 51 61 55 1 59 1 64 2 63 3 34 3 35 3 37 2 59 3 56 2 56 1 3 2 35 2 49 27 1 41 3 534 1 52 54 42 41 2 58 1 64 3 64 1 1 7 25 6 6 4 39 35 4 28 2 28 7 28 4 28 9 28 6 1 11 28 15 33 1 3 4 5 1 33 3 55 2 45 28 1 7 3 62 2 7 1 4 1 63 1 26 2 24 3 7 4 28 18 33 2 1 925 9 28 5 28 24 28 14 25 3 1 1 43 25 1 58 4 28 19 28 3 25 2 1 1 1 8 28 16 2 4 6 8 1 12 Figure 4. Proportion of bass anticipations against mean event rate per piece. Zero proportions (34 pieces) were excluded from the calculation of the regression line. Magaloff s performance. There is also an effect of event rate, suggesting that bass leads become less frequent as the tempo of the pieces increases (Figure 4). Again, slower pieces leave more room for expressive freedom than do faster pieces. To further analyze the occurrences of bass anticipations, we categorized all score events bar-wise into first beats, on-beats (all beat events except the first beat), and off-beats. It turns out that metrical position has a significant effect: the highest number of bass anticipations fall on the first beat (1.8%); other on-beat events receive 1.48% bass anticipations, and.66% are found on off-beat events. This suggests that bass anticipations are used by Magaloff to emphasize predominantly strong beats. 4.3 The Earlier Type of Tempo Rubato An expressive means that has a long performance tradition is the tempo rubato in the earlier meaning [3]. It refers to expressive temporal deviations of the melody line, while the accompaniment, offering the temporal reference frame, remains strictly in time. Chopin in particular often recommended his pupils to keep the accompaniment undisturbed like a conductor, and give the right hand the freedom of expression with fluctuations of speed [3, p. 193]. In contrast, the later meaning of tempo rubato was used more and more to refer to the parallel slowing down and speeding up of the entire music (today more generally referred to as expressive timing). In expressive performance, both forms of rubato are present simultaneously. We aim at identifying sequences of earlier tempo rubato automatically from the entire corpus. To extract overall information about sequences where Magaloff apparently employed an earlier tempo rubato, we count the out-of-sync re- Page 294

Number of O o S Regions 1 8 6 4 2 48 2 32 2 r=.349*** n=89 p<. 58 317 4 9 2 62 1 55 2 6 27 2 48 1 28 17 56 3 26 1 5 2 9 1 35 2 519 3 15 1 28 432 1 55 1 1 3 3 3 63 2 34 2 57 3835 1 2 54 1 37 1 1 6 4 3 24 1 37 2 6 27 1 59 1 49 3652 23 28 2 28 15 25 7 63 3 3 1 33 4 62 2 15 2 24 4 26 2 41 4 47 6146 58 1 19 29 25 1 25 2 35 3 15 3 28 13 33 1 17 1 45 7 3 3 2 24 3 5 3 4 2 25 1 41 2 64 3 1 9 31 539 28 5 28 1 1 12 58 4 25 11 35 4 28 2 28 7 28 9 28 6 1 11 41 1 17 217 3 3 4 5 1 33 3 28 1 59 259 3 24 2 6 3 6 1 4 1 7 1 63 1 6 4 41 3 28 21 44 56 2 56 1 28 11 53 7 4 28 18 34 1 25 4 7 5 64 2 12 28 228 23 33 2 25 9 34 3 25 8 28 12 28 24 28 14 16 4 4 25 3 1 7 42 1 1 25 6 58 2 64 1 43 1 21 1 28 19 28 3 1 4 1 5 1 8 28 16 2 4 6 8 1 12 Figure 5. The number of out-of-sync regions (earlier tempo rubato) per piece is plotted against the event rate. gions of each piece. An out-of-sync region is defined as a sequence of consecutive asynchronies that are larger than the typical perceptual threshold (3 ms) and that contain more elements (events) than the average event rate of that piece (2 13 tones). Usually faster pieces contain many shorter runs that are out-of-sync, but due to the fast tempo, these regions extend only to some fraction of a second. To take this into account, the search for out-of-sync regions is coupled to the piece-wise average event rate. On average, a piece contains 1.8 such regions. The pieces with the lowest numbers are generally the Mazurkas, Preludes, and Etudes (below 1), the pieces with the highest counts are by far the Nocturnes (on average well over 5), suggesting that particularly this genre within Chopin s music leaves most room for letting the melody move freely above the accompaniment. Figure 5 shows the number of out-of-sync regions per piece against the average event rate of the piece. It demonstrates that faster pieces contain fewer such regions, suggesting that this form of tempo rubato is bound to slower and medium tempi (such as the Nocturnes, the slowest piece category in the Chopin corpus). This overall finding is not surprising; the earlier tempo rubato is expected to be more often found in melodic contexts rather than in virtuoso pieces, as the historic origins of the earlier tempo rubato go back to vocal music. To illustrate, the example of the visualization tool presented in Figure 1 is briefly discussed. It shows an excerpt (bars 5 54) of the Nocturne Op. 27 No. 2. This example contains two runs of tempo rubato as determined by the algorithm (indicated by horizontal bars in the middle panel). The first starts on the downbeat of bar 5, where Magaloff delayed the melody note by 265 ms, only to be early over the next few notes of the descending triplet passage. The beginning of the 48-tuplet figure (which is interpreted as 1/16-note triplets as well) also leads the accompaniment. Towards its end, the second run of tempo rubato as determined by our algorithm begins, just when Magaloff starts to lag behind the accompaniment. This lag coincides with a downward motion and a notated decrescendo. The following embellishment of the b-flat (notated as 1/32 notes and 1/32 triplets) is again clearly ahead of the accompaniment. The first note of the next phrase is also ahead, potentially to underline the notated anticipation of the upcoming harmony change towards e-flat minor. 5 SUMMARY AND FUTURE WORK This paper presented a computational approach to making large performance corpora accessible for detailed analysis. It defined and automatically measured between-hand synchronization in over 15 pieces by Frédéric Chopin. Working with data sets of that size, i.e., complete sets of performances of a single composer or several hundred thousand played notes requires, among other things, effective scoreperformance matching algorithms and interactive graphical user interfaces for post-hoc data inspection and correction. Preliminary data analysis of the between-hand synchronization attempted to demonstrate the rich use of asynchronies in Magaloff s complete Chopin corpus, a historic document of a unique performance project that offers unequaled opportunities for performance analysis. We sketched overall trends of asynchronicity with respect to pieces, tempo, and metrical constraints, as well as specific cases of bass anticipations and occurrences of tempo rubato in its earlier meaning. This research endeavor is preliminary as it stands. It is meant to lead to further modeling efforts to be able to predict asynchronies from Romantic scores following Nikita Magaloff s instrinsic style. Piece category, overall musical texture, as well as local and global aspects from the scores may be promising features for training machine learning algorithms in order to derive predictive computational models of between-hand asynchronization. Such models may enhance existing performance rendering systems by adding an important expressive feature. To be able to automatically assess performance corpora of this scale offers completely new pathways for computational musicology. Historic documents such as the present corpus are in managable reach for detailed analysis. Other large corpora, such as piano rolls of historic reproducing pianos or the performance database of the Yamaha ecompetition 4, may be other sources for future large-scale performance investigation. Finally, detailed knowledge derived from performances by established musicians may help us develop real-time visualization tools that give intelligent feedback to practicing piano students to enhance their awareness of what they are doing, and potentially help them to improve their play instantly. 4 http://www.piano-e-competition.com Page 295

6 ACKNOWLEDGEMENTS This research was supported by the Austrian Research Fund (FWF, grant P19349-N15). We are indebted to Mme. Irène Magaloff for her generous permission to use her late husband s performance data for our research. 7 REFERENCES [1] GOEBL, W. Melody lead in piano performance: Expressive device or artifact? Journal of the Acoustical Society of America 11, 1 (21), 563 572. [2] GOEBL, W., AND PARNCUTT, R. The influence of relative intensity on the perception of onset asynchronies. In Proceedings of the 7th International Conference on Music Perception and Cognition, Sydney (ICMPC7), Aug. 17 21, 22 (Adelaide, 22), C. Stevens, D. Burnham, G. McPherson, E. Schubert, and J. Renwick, Eds., Causal Productions, pp. 613 616. [3] HUDSON, R. Stolen Time: The History of Tempo Rubato. Clarendon Press, Oxford, 1994. [4] MONGEAU, M., AND SANKOFF, D. Comparison of musical sequences. Computers and the Humanities 24 (199), 161 175. [5] VERNON, L. N. Synchronization of chords in artistic piano music. In Objective Analysis of Musical Performance, C. E. Seashore, Ed., vol. IV of Studies in the Psychology of Music. University Press, Iowa City, 1936, pp. 36 345. Page 296