Dynamics and Relativity: Practical Implications of Dynamic Markings in the Score

Transcription

1 Dynamics and Relativity: Practical Imlications o Dynamic Markings in the Score Katerina Kosta 1, Oscar F. Bandtlow 2, Elaine Chew 1 1. Centre or Digital Music, School o Electronic Engineering and Comuter Science, Queen Mary University o London 2. School o Mathematical Sciences, Queen Mary University o London abstract This article ocuses on the meaning and ractical maniestations o exressive markings in the music score, seciically those markings that corresond to loudness levels such as (iano), m (mezzo orte), and (ortissimo). We resent results showing how the absolute meanings o dynamic markings change, deending on the intended (score deined) and rojected (actually erormed) dynamic levels o the surrounding musical context. The analysis o recorded erormances shows dierent realisations o the same dynamic markings throughout a recording o a iece o music. Reasons or this henomenon include the score location o the markings, such as the beginning o a iece, and the marking s location in relation to that o revious ones. Observations show that more oten than not, the transition between the two markings is more consistent when ianists move rom a louder to a soter marking, or move between markings that both reresent a high intensity, or when the markings show high levels o contrast. For markings that aear in the score more than twice, there tends to be signiicant dierence in the ways they are interreted in a recording. Finally we resent evidence that ianists with high numbers o outlier recordings, i.e. showing uniqueness in the ways they transition between dynamic markings, tend to co-occur in outlier clusters or a signiicant number o Mazurkas, meaning they diverge rom common ractice in similar ways. Alications o these results include exression synthesis and music transcrition. 1 Introduction Musical exression is an imortant art o music erormance analysis. Palmer, in [29], underscores that erormers maniulate the sound roerties, including requency (itch), time, amlitude, and timbre (harmonic sectrum) above and beyond the itch and duration categories that are determined by comosers. These maniulations deine the term musical exression which is artially achieved by altering the variables or stress, rhythm, accent, and intensity contour or the uroses o communicating emotion and clariying structure. Much research has ocused on analysing and modelling the correlation between these acoustic roerties, examles include studies on the relationshi between temo and loudness (e.g. [37] and reerences therein) or itch and loudness (e.g. [14] and reerence therein). However, less attention has been aid to the relation between such roerties and their reresentation, as a means o communication between the comoser and the erormer. Our study ocuses on musical exressivity and aims to understand the connection between loudness levels in recordings and dynamic markings as notated in the score. The roblem o music interretation is a comlex one. The score is a reresentation o the comoser s intentions. The same symbol be it a note, dynamic marking, indication o articulation, or hrase grouing can have a variety o ossible interretations. The original score is reracted through the erormer [31,.59], who can choose to render the symbols in unique ways. The audio eect o the reracted work is received by the listener, who in 1

2 turn may erceive it through their own individual lenses. This aer ocuses on two o these ersectives, that o the comoser (as communicated by a seciic score edition) and that o the roessional erormer (as archived in a recording). Cook in [7,.14] highlights that in books on classical music there is a distinction between authors, i.e. comosers and reroducers, i.e. erormers; The latter ones used to be mentioned when they obscured the original music through over-interretation or gratuitous virtuosity. In the case o Choin, being both comoser and erormer, his ersonal erorming style was imitated internationally, although it was not his intention to emhasize his originality [10,. 215]. The above has to be taken into account when we analyse dierent interretations o his ieces. As many o the erormers might imitate his style, we should acknowledge that in terms o dynamics all his contemoraries agree in reorting that his dynamics did not exceed the degree o orte, without however losing a single bit o shading [10,. 215]. Dynamic levels in musical exressivity are reresented in the score by markings such as (iano, meaning sot), and (orte, meaning loud). These symbols are interreted in erormance and communicated through the varying o loudness levels. I we were to order the set o dynamic markings in increasing loudness, yielding the ollowing sequence, termed Ordinal Loudness Sequence (OLS) in the ollowing, < < m < m < <, (1) we would assume that the loudness level corresonding to a low-rank marking in the OLS is lower than the level corresonding to a higher-rank marking in the OLS. As we shall see this is not the case. The exected loudness o dynamic markings may be suerseded by their local context so that a dynamic might objectively be louder than a at another art o the iece. The connection between loudness levels in erormance and the rinted dynamic markings in the score is still oorly understood. While the meaning o rosodic inlections given the same words have been widely studied in seech and linguistics 1, very little work exists that addresses dierent meanings o the same symbolic reresentations o exressive nuances in music. The goal o our study is to better understand the arameters which deine the erormers resonses to dynamic markings. Surred on by our irst study in [23], this article resents an exanded and deeer analysis o the nature o the relativity in exression o dynamic levels. In this study we examine the dynamic markings in the OLS and we consider two tyes o relativity in the interretation o these markings, one dealing with the overall loudness level o a articular marking throughout the times that is resent in the score o one iece, and the other dealing with the distinct loudness level o a articular marking at a single osition in the score. In order to gather the inormation we need or the irst tye o relativity, or each iece, we comute the average loudness levels over all recordings or all markings that are the same, and comare the resulting averages. About the second tye o relativity, or each iece, we comute the change in loudness or each air o dierent consecutive markings. For examle, assume that a iece has the sequence o markings ( 1, 2, 1,, 3, 2 ), in 1 For examle, studies have examined the meanings o nuances in utterances o the words whatever [4] and okay [20]. 2

3 the order as they aear in the score. Then or each erormance recording we get the loudness values that corresond to each o the markings. For the irst tye o relativity, each recording has a loudness level o the s which is comuted by taking the average o the loudness values that corresond to the three ositions, and the same rocess is ollowed or all the remaining markings. Then these average values are comared (more details about the comarison method in Section 4.1.) For the second tye o relativity, or each recording we comare the loudness change between the dierent consecutive markings in airs ( 2, 1 ), ( 1, ), (, 3 ), and ( 3, 2 ), individually. Additionally we analyse how the distance o the markings in OLS aects the transition rom one loudness level to another, and inally we observe the dierent atterns o the maniestation o the same markings as they aear in a score sequence o one iece. More recisely, we seek to answer the ollowing questions: (i) Do the aggregate dynamics o airs o dynamic markings conorm to the exected order? (ii) Do airs o individual instances o dynamics at markings conorm to the exected order? (iii) How does the average dynamic change vary related to the ordinal distance between dynamic markings? (iv) How are the same dynamic markings realised as they recur throughout the iece? Along the direction o analysing general dynamic change behaviour throughout a iece, we also conduct an exeriment on analysing seciic atterns that emerge on the way dierent ianists maniulate dynamic changes over time based on the ositions o the dynamic markings, with a ocus on excetional interretations. A challenge o conducting systematic research on exressive arameters is the lack o audio data and annotations or analysis and or training. In order to understand the range o exressive ossibilities or each dynamic marking, we required a signiicant number o recordings or each iece. For this study, we have drawn rom the CHARM 2 Choin Mazurka dataset, which consists o known recordings o Choin s Mazurkas amassed or the urose o studying erormance styles. For the majority o ianists, the romantic reertoire remains the chie resource or exressive ossibilities [3]. Hence, our choice to ocus on Choin s Mazurkas. In order to address the our questions raised above, we have created a score-beat loudness maing rom the data, that is, we have matched ositions o score markings with their corresonding loudness levels in recordings. In order to obtain reliable measurements and scalable analyses, we rely on comutational audio analysis tools, which desite their imerections are becoming art o the standard equiment or emirical musicologists [12, ]. To seed u the labour-intensive rocess o annotating beat ositions or each recording, we develoed a heuristic or automating the alignment o multile audio iles. Loudness values are extracted and analysed using standard audio rocessing and statistical techniques. 2 htt:// accessed January

4 This research will enable the building o automatic methods that can more accurately and meaningully ma musical exression in recorded music to an ontology or music dynamics, beyond simly extracting a direct dynamic level; such techniques will also be valuable or music transcrition. On the synthesis side, the results can be a useul tool or generating exressive renderings o notated scores. However, one must take into account two main considerations concerning the score markings: irstly, the dynamic markings may simly reinorce the natural redisosition o a music art to be exressed in a articular way based on its structure; secondly, recise annotation by the comoser can be resent at a nonobvious lace [18]. Together with the large amount o inormation and erormance related arameters [5], the challenges above make the modelling o music exression one o today s most imortant unsolved roblems in the descrition o music audio. The article is organized as ollows: Section 2 describes related work; Section 3 describes the creation o the dataset or the studies o this article; Section 4 resents the results o the studies; Section 5 resents the analysis o clustering results between recordings; ollowed by conclusions and discussions in Sectiond 6 and 7, resectively. 2 Related research Extremely little work exists that examines the maing rom score to audio loudness and vice versa, an excetion being our reliminary study on the meanings o dynamic markings in erormances o ive Choin Mazurkas [23]. [15] resents an early study on dynamic contrasts in relation to score dynamic markings. This study analysed data samled rom 60 commercial recordings o choral, orchestral, and iano comositions mostly rom the 19th century and rom comosers including Beethoven, Choin, and Schumann, amongst others. The dynamic changes within the music samles were recorded using a Bruel and Kjaer Grahic Level Recorder, which enables the continuous measurement o relative intensity changes. Across the music excerts samled, the results showed a larger dynamic range rom to, with an average change o 13.42dB, than rom to, with an average change o 11.97dB. [32] roosed the MUDELD (MUsic Dynamics Extraction through Linguistic Descrition) algorithm, which uses linguistic descrition techniques to categorise dynamic labels o searate musical hrases into three levels designated as iano (), mezzo (m), and orte ( ). The basic idea o this study was to extract loudness inormation rom audio inut by comuting the Root Mean Square o the signal ollowed by normalisation so that the resulting values lie in [0, 1]. The music signal was then manually segmented, each segment was then reresented by its average loudness value. Next, disagreement values were comuted or the distance between the segment s value label and the reerence label obtained rom the score. As an initial exeriment, eight commercial recordings o Debussy s Syrinx were retrieved and manually segmented into hrases. The outcome o the exeriment indicates that the resulting labels were similar to the score indications. In [18], a linear basis ramework (LBM) is roosed to account or exressive variations as well as the eect o exressive markings in the score on music erormances. The aroach relies on the creation o a number o hand crated numerical descritors, or basis unctions, encoding certain structural asects o the score. Each o the basis unctions is linked to one 4

5 score marking and reresents the activation o that marking on a scale o 0 (non-active) to 1 (active). The basis unctions serve as indicators or note attributes, such as stacatto or exressive markings, or gradual (e.g. crescendo) or instant (e.g. iano) changes. Essentially, LBM rovides a way to determine the otimal inluence o each o the sets o basis unctions with a set o weights that aroximate the imact o an exressive arameter. To test the LBM aroach, two exeriments were erormed; the irst examined how accurately exressive dynamics can be reresented and redicted using as dataset the Magalo corus [13], recordings o Choin s ieces erormed on a Bösendorer recording iano. The results showed that, or both reresentation and rediction, the correlation ranges rom weak to medium or various combinations o basis unctions, as well as or weights comuted globally (same in all ieces) or locally (dierent or each iece). The latter roduced better results. Also, rediction variance was substantially lower than the variance observed. The second exeriment examined how well the LBM ramework reveals dierences between erormers using loudness curves o commercial recordings o various ieces layed by dierent ianists. The results showed that across ieces the variance o coeicients rom the basis unctions is too large to make direct links between the coeicients and individual erormances, indeendent o the iece. This inding resonates with our results in [24]. The basis ramework has been used in the case o recorded orchestral music in [19] where the measure o dynamics is the overall variation o loudness over time. Also, a study o a machine learning aroach concerned with the score-based rediction o searate note intensities in erormed music was resented in [17]. Our aroach diers rom rior work in its ocus on loudness reresentation (notated loudness) and the degree to which the reresentations cature constant and relative dynamic changes as realised in erormance. 3 Data Prearation In this section we resent the data that has been created and used or the urose o this study. More seciically, Section 3.1 describes the dataset, and Section 3.2 describes the loudness arameter that orms the basis o our analysis. 3.1 MazurkaBL Dataset For this and related studies, we have created the MazurkaBL dataset [21], a collection o score-beat ositions and loudness values, with corresonding score dynamic and temo markings or 2000 recordings o orty-our Choin Mazurkas. The dataset is ublicly available and it can be ound in [27]. The audio iles used or the creation o this dataset are derived rom the Mazurka Dataset 3 which oers a signiicant number o dierent interretations o the same Mazurkas by Choin as erormed by dierent roessional ianists. MazurkaBL ocuses on a art o the dataset containing 2000 audio recordings o erormances o ortyour ieces. The inal number o recordings selected or each Mazurka is shown in Table 1. The audio recordings included in MazurkaBL are the ones that adhere to the reetition instructions in the score; excessively noisy recordings have been excluded. 3 htt:// accessed November

6 Mazurka index M06-1 M06-2 M06-3 M07-1 M07-2 M07-3 M17-1 M17-2 M17-3 M17-4 M24-1 # recordings Mazurka index M24-2 M24-3 M24-4 M30-1 M30-2 M30-3 M30-4 M33-1 M33-2 M33-3 M33-4 # recordings Mazurka index M41-1 M41-2 M41-3 M41-4 M50-1 M50-2 M50-3 M56-1 M56-2 M56-3 M59-1 # recordings Mazurka index M59-2 M59-3 M63-1 M63-3 M67-1 M67-2 M67-3 M67-4 M68-1 M68-2 M68-3 # recordings Table 1: Choin Mazurkas used in this study, and the number o recordings, and the number o dynamic markings that aear in each Mazurka; the Mazurkas are indexed as M<ous>-<number>. For our analysis, we ocus on the ollowing markings: S = {,, m,, }. (2) occurs 63 times, 234, m 21, 169, and 43 times in the dataset, giving a total o 530 markings. The number o markings that aear in each iece individually is shown in Table 1. In the next subsections we exlain how we link the beat ositions in the score with corresonding ones in each recording, and we describe how we comute the loudness inormation or each dynamic marking. For simlicity, we ocus on the loudness values that corresond to the marking ositions, while acknowledging the imortance o the dynamic changes in between. The way the data is structured allows us to create a score-based inormation model. The recording dates ranging rom 1902 to the early 2000 s. Furthermore, the score edition used by each erormer is not known. As mentioned in [1,.56], since most o his [Choin s] works were ublished in simultaneous irst editions in France, Germany and England, and since he also made alterations in the scores o various uils, there are inevitably many discreancies. Thus, the many ianists could be laying rom rather dierent editions. However, tracing the actual score used in the rearation o each erormance is unrealistic. For the uroses o obtaining score-based dynamic markings or this study, we used the edition by Paderewski, Bronarski and Turczynski, as it is one o the most oular and readily available editions. A general overview o dierences among score editions on dynamic markings in S reveals the ollowing: a oular case is that a marking might be located in a dierent osition by one or two beat distance, a non-oular case where a marking is either missed or is additional, a non-oular case where a marking is resented inside a arenthesis while it does not aear in other editions, and a rare case o a marking exressed as a dierent marking elsewhere. We created an XML version o the scores and inormation concerning the exact location o each dynamic marking was extracted automatically using Music21 4 [9]. 4 web.mit.edu/music21, accessed January

7 3.2 Loudness inormation or markings We have already mentioned in the introduction that we ocus on audio eatures that contain inormation about loudness. More seciically we would like to cature the change in dynamics through a iece o music. Laboratory attemts to measure loudness in terms o how much louder one sound is than another have used estimations o various loudness distances or ratios. A sone scale o loudness, based on estimates o aarent ratios rovides a way to describe aarent loudness relationshis [2]. In our research we have used sone measurements to quantiy the loudness o audio signals. An advantage o the sone scale is that it is linear: doubling the sone values corresonds to doubling the loudness sensation o a corresonding tone, allowing or more accurate normalisation. Another equally imortant reason is that we wish to train our models on audio that is re-rocessed based on the basic sychoacoustic concet o equal loudness curves, and not to make any other comression or modiication. I a machine were to listen like a human, then it needs an inut similar to that o a listener. For the exeriments resented in this article, loudness inormation is extracted rom the audio signal using the ma_sone unction in Elias Pamalk s Music Analysis toolbox 5. The seciic loudness sensation in sones er critical band is calculated by ollowing the rocess exlained in [30]. Using this rocedure, we calculate the ower sectrum o the audio signal using a Fast Fourier Transorm. We then use a window size o 256 samles, a hosize o 128, and a Hanning window with 50% overla. The requencies are bundled into 20 critical bands and these requency bands relect characteristics o the human auditory system, in articular o the cochlea in the inner ear. [30] We calculate the Sectral masking eects using the method described in [33]. Then, we calculate the loudness as db-spl units; rom these values we calculate the equal loudness levels in Phon, and convert the values to sones ollowing the ormula described in [6]: { 2 (L 40)/10, P 40 S = (3) (L/40) 2.642, P < 40, where L is the loudness level in hons, S is the loudness level in sones, P is the absolute db value o the enveloe exressed in hons via stored curves o equal loudness levels. This calculation accounts or the threshold o hearing and the ear s nonlinear and requencydeendent resonse to intensity dierences. [6] This re-rocessing stage transorms each recording into a dynamic time series (Fig. 1- to, blue curve). The sone values are smoothed by local regression using weighted linear least squares and a 2 nd degree olynomial model (the loess method o MATLAB s smooth unction 6 ) (Fig. 1 to, brown curve). From the smoothed data, we consider the values o the score-beat ositions (yellow x s in Fig. 1 to), which constitutes a sequence x n, n N, where n is the number o score beats in one iece. The sequence is then normalised to [0, 1] by dividing the values by the maximum loudness value o the articular recording. In this way we are able to comare dierent recording environments which serves our urose o ocusing on relative rather than absolute changes. 5 accessed 20 February uk.mathworks.com/hel/curveit/smooth.html?reresh=true, accessed 3 January

8 For each Mazurka, the irst author manually annotated the beats or one recording, and beat ositions were transerred automatically to the remaining recordings using a multile erormance alignment heuristic. The alignment uses the algorithm described in [11], which is based on Dynamic Time Waring using chroma eatures; the heuristic otimises the choice o reerence recording. A characteristic o Choin s music is that it draws insiration rom singing, which translates to a style o iano laying whereby the melody may be dislaced rom the corresonding beat osition in the accomaniment so as to convey luidity o exressive timing. Related to this, see [16] on the melody lead eect. As a rule, in our manual annotations, we have chosen to ollow the melody line so as to cature the lyricism o the rubato. The markings that are analysed are marked in Fig. 1 bottom as black x s. The loudness value corresonding to each marking is the average o that ound at the beat o the marking and the two consecutive beats, excetions being the markings where in the ollowing two beats there was no event; in such cases we considered only the loundess value that corresond to the current beat. More ormally, i {y n } R is the sequence o the normalised loudness values in sones or each score beat indexed n N in one iece, then the loudness value associated with the marking at beat b is l b = y b+(i 1). (4) i=1 The reason behind choosing three beats is based on the observation that sometimes the actual change in resonse to a new dynamic marking does not take lace immediately, and can only be observed in the subsequent beat or two. It is clear rom the data that loudness varies considerably between one dynamic marking and the next. Thus, we additionally aim to have the smallest window ossible to cature dynamic changes in resonse to a marking. Consequently, we have chosen a three-beat window as the window or study, which or Mazurkas corresonds to a bar o music. 4 Perormed Loudness Study 4.1 Ordinal Loudness Sequence Preserved On Average? In this section we exlore the general behaviour o ianists resonses to the dynamic markings in the set S. More seciically, this section addresses the issue o whether the dynamic level o a soter marking could be higher on average in a recording than that o a louder marking. We deine the exected rank order sequence or loudness behaviour, and use Kendall s tau rank correlation coeicient test to show that the average dynamic levels in each recording do not always abide by the exected rank ordering. In a recording, each dynamic marking is reresented by a dynamic value (in sones), as described in Section 3. As some markings aear in the score more than once, we comute the aggregate value or each marking in a recording by taking the average o the loudness values whenever the same marking aears in the score: or a symbol s S that aears n 8

9 18 Mazurka O. 6 No. 1 (ianist: Csalog, 1996) Sones Samles Score beat Figure 1: Loudness reresentation o a single Mazurka O. 6, no. 1 recording (ianist: Csalog, recording year: 1996). To: Smoothed curve o raw data o sone values; the score beat ositions are highlighted by x s. Bottom: Normalized score beats curve; the averaged inal loudness values or each marking are highlighted by x s. 9

10 times in the score at the beat ositions {s 1, s 2,..., s n }, E(s) = 1 n n l si. (5) i=1 We irst consider or each recording the average loudness levels E(), E(), E(m), E(m), E(), and E() or the markings,, m, m,, and, resectively. It so haens that m does not occur in the scores we consider in this study. For each recording in a articular iece, each unique air o markings is associated with their resective average erormed loudness. Each air o these values is then comared to the exected ordering o the markings according to the OLS. A Kendall s τ value o 1 indicates erect rank agreement with the OLS, and a value o 1 indicates the converse. For each iece we comute the mean o τ value, each corresonding to a articular marking air and derived rom a recording o that iece. The results o the mean Kendall s tau rank correlation coeicient test are summarized in Table 2. The numbers in arentheses in Table 2 are the results o the mean Kendall s tau test ollowing the same rocess as beore. However, now the loudness level er marking is not derived rom a three-beat window size, instead it is the average o all beats rom the osition o the marking until a new marking or indicator or dynamic change aears in the score. The comarison o the values highlights the excetions to the OLS in Mazurka O. 33 No. 4 where is soter than in most instances when one considers the entire duration o the marking, Mazurka O. 68 No. 3 where is louder than in most cases i we use the bigger window, Mazurkas O. 50 No. 1 and O. 56 No. 3 where m is louder than, as well as Mazurka O. 67 No. 1 where is soter than m. The case o (osthumous) Mazurka O. 67 No. 2 is o articular interest as the interretation o all the markings does not adhere to the OLS in most cases. The remaining negative mean τ values, highlighted in bold (and red), are urther investigated and analysed in the Aendix. In summary, the case studies have shown that the imortant actors are the relative loudness o neighbouring markings, the inter-relations o nearby markings and other score inormation, the structural location o the markings, the local shaing o loudness, and generally the creative license o the erormer. The results also suggest that how well the use o loudness comlies with the OLS is related not so much to the number o dierent markings resent, but rather the local context and memory. As general conclusion, we cannot assign ixed thresholds or the dierent dynamic markings. In order to deeen the understanding o the way in which markings are exressed, the next section ocuses on air-wise marking comarisons as reresented by dynamic change values or each air o consecutive markings. 10

11 Pair Mazurka {,} {,m } {, } {, } {,m } {, } {, } {m, } {, } M (1) 0.94(1) 1(1) 0.35(0.77) 1(1) 1(0.88) M06-2 1(1) M (0.95) 1(1) 1(1) 1(1) 1(1) 0.71(0.57) M (1) 1(1) 1(1) 1(0.81) 1(1) 1(1) M07-2 1(1) M07-3 1(0.97) 1(1) 1(1) 0.97(1) 0.97(0.97) 0.83(0.69) M (1) M (0.68) 0.88(0.72) 0.96(0.64) M (0.94) M (0.67) 1(0.97) 1(0.88) M24-1 1(1) 1(1) 0.70(1) M (1) 0.96(1) 0.54(0.95) M (0.85) M (0.96) 1(1) 1(1) 0.93(0.96) 1(1) 1(0.96) M30-1 1(0.96) M (0.88) M (0.89) 1(1) 1(1) 0.96(0.82) 0.82(0.74) -0.33(-0.30) M (0.96) 1(1) 1(1) 1(1) 1(1) 0.93(0.20) M (0.96) M33-2 1(1) 1(1) 0.96(0.80) M (1) M (0.97) 0.81(1) 1(1) M (0.94) 1(1) 1(1) 1(1) 1(1) 1(1) M (0.81) -0.19(0.34) 1(0.62) -0.48(-0.81) 1(0.62) M (1) 1(0.74) 0.95(0.95) M (0.33) 1(0.58) 1(0.97) 0.88(0.27) 1(0.94) 0.94(0.76) M (0.42) 0.91(1) 0.91(0.82) M (0.20) M (0.34) 1(1) -0.67(-0.91) 1(0.96) -0.94(-1) -1(-1) M56-1 1(0.94) 1(0.94) 0.18(0.29) M (0.96) M (0.10) 1(1) 0.96(0.96) M59-1 1(1) M59-2 1(1) 1(1) 1(1) 0.96(1) 1(1) 0.89(1) M59-3 1(1) M (1) 1(1) 0.95(1) M63-3 1(0.61) M (0.94) -0.43(0.37) 0.83(0.94) 0.94(1) -0.89(-0.77) 0.09(0.49) 0.43(0.66) 0.94(0.77) 0.71(0.49) M (-0.55)-0.29(-0.42)0.03(-0.23) 0.42(-0.30) 0.61(-0.09) 0.23(-0.03) M (-0.9) 0.45(0.65) 0.90(0.80) 1(1) 1(1) 0.90(0.90) M (0.81) 0.95(0.90) 0(0) M (1) M (-0.02) 0.96(0.96) 1(1) 1(0.91) 1(1) 1(1) M (0.95) 1(1) -0.10(0.86) Table 2: Mean Kendall s τ values or airwise comarisons o the average loudness values that corresond to the dynamic markings or each recording and the corresonding values o the markings at the OLS: values o 3-beat window (marking window). Negative values are highlighted in bold and coloured red. 11

12 4.2 Is the Ordinal Loudness Sequence Preserved In Pairwise Instances? In this section we examine individual resonses to consecutive airs o distinct dynamic markings. We investigate consecutive resonses to markings, or examle, (, ) and (, ), that indicate an ascending or descending dynamic change, highlighting results contrary to the OLS, meaning that the recordings ollow the loudness change rom one dynamic marking to another in the order as it is deined in Equation 1. In this section, we consider only airs o distinct consecutive markings. Fig. 2 shows the log loudness ratios or airs o distinct consecutive dynamic markings throughout the orty-our Mazurkas. For each air o dynamic markings, the agreement with the OLS may vary deending on dierent exression strategies. Suose the loudness o the (k 1)-th dynamic marking, say a, is exressed by a ianist as l k 1, and the loudness o the k-th dynamic marking, say a, is exressed as l k, then the log ratio (as summarized in Fig. 2) is log( l k l k 1 ). The data is ound to be more easily comared with the log reresentation, as the result can be ositive or negative deending on moving rom a bigger value to a smaller one or the oosite, resectively. Observe that there are cases where the mean value o the ratios in the to (let) lot is lower than 0, meaning that a good number o recordings do not resect the OLS at those seciic markings. Subsequently, there are cases where the mean value o the ratios in the bottom (right) lot is higher than 0, meaning that good number o recordings do not adhere to the OLS at those seciic markings. In Table 3 we show the Mazurkas that have marking airs in which the average loudness change contradicts the OLS. These corresond to marking airs with below 0 log ratios in the to (let) lot and those with above 0 log ratios in the bottom (right) lot o Fig. 2. In the arentheses we resent the roortion o the outlier airs over all consecutive marking airs resent in that seciic Mazurka. We do not consider the sequential airs that consist o the same marking. Outlier in ascending air Mazurka (roortion o outlier airs) (,) M33-4 (0.25) M41-1 (0.25) M67-3 (0.11) M68-2 (0.05) (,m ) M50-1 (0.17) (m, ) M41-2 (0.75) (, ) M41-2 (0.75) M50-1 (0.17) M50-2 (0.14) M67-1 (0.09) Outlier in descending air Mazurka (roortion o outlier airs) (,) M17-2 (0.25) M50-3 (0.08) M67-2 (0.14) (,m ) M41-2 (0.75) Table 3: List o marking airs that contradict the OLS with inormation on Mazurkas where the airs aear as well as their roortion with resect to the total number o airs in that Mazurka. Mazurka O. 41. No. 2 has the largest roortion o outlier airs. More seciically, the sequence o the markings in this Mazurka is {,, m,, }, and in all transitions excet 12

13 Figure 2: The log ratio o the loudness in the transition rom a marking m k 1 to a marking m k in the score sequence (m k 1, m k ) where l k 1 < l k (to-let) and l k < l k 1 (bottom-right) ollowing the OLS. 13

14 or the last one, the average loudness change ratio over all recordings contradicts the OLS. The unredictability o the loudness rogressions is evidenced in the drastic variations in the loudness curves o the recordings o this Mazurka as resented in Fig.?? in Section 4.1. Moving on, Fig. 3 shows the average standard deviation o the log loudness ratios or dierent marking airs over all Mazurkas. One would exect that, as the distance between markings in the OLS increases, the ratio o the loudness change would also increase. But, this is not the case, as illustrated by the ollowing observations. In all cases excet or the airs (, ) and (m, ) the average standard deviation (SD) is larger in the ascending direction than the descending one. This means that, more oten than not, the transition between the two markings is more consistent when ianists move rom a louder to a soter marking. Note that the ive smallest SD values can be ound or the sequences (m, ), (, ), (, ), (, ), and (, ) with corresonding values , , , , and This means that the dynamic change is more consistent or airs o markings near the uer limit o the OLS uer limit and at the extremes o the OLS (those having the greatest distance on the OLS scale). Average o standard deviation er marking air Second marking m m First marking Figure 3: The average standard deviation o the log loudness ratio or each marking transitions are rom the x-axis to the y-axis. The above observations relate to loudness transitions between consecutive and distinct markings, thus dealing with local behaviour. Having considered the average behaviour o individual markings in the revious section, the next section will study the evolution o these individual markings over the course o a erormance o a iece. 14

15 4.3 Analysis o dierent maniestations o the same dynamic markings throughout a iece We have already observed rom the revious indings that the dynamic markings are not erormed in the same way when they aear in the score more than once. In this section, we answer the question: How dierent is the erormance o the same dynamic marking over the course o a iece? In resonse to this question, we comare the dynamic values corresonding to the same marking whenever it aears, using two kinds o Analysis o Variance (ANOVA). A one-way ANOVA is used to comare the means o the dynamic values whenever a seciic dynamic marking aears in the score more than once, while a two-way ANOVA is used to comare the means o the dynamic values whenever a seciic dynamic marking aears in the score more than once, and the means o each erormer s resonse over the observations. Both ANOVA tyes test the hyothesis that the data samles are drawn rom oulations that do not have the same mean against the null hyothesis that the oulation means are the same. A result is statistically signiicant i it allows us to reject the null hyothesis. In order to measure statistical signiicance o the results, we use the P value which gauges how well the data suorts the null hyothesis. A low P value suggests that the samle rovides enough evidence that the null hyothesis, which is that the oulation means are the same, can be rejected. It is worth noting that, in deense o our use o the P value, we have merely used the P value to determine i there is an eect, and not to quantiy the eect [28]. In the uture, the eect size and the conidence interval can be used to convey the magnitude and the relative imortance o the eect; this is currently out o scoe o this article. The two sets o ANOVA results are resented in the P value 1 and the P value 2 columns resectively in Table 4. P values less than the chosen signiicance level α = 0.05 are highlighted in bold (and coloured red). P > 0.05 means that there is no signiicant dierence between the dynamic values o the same markings. It is observed that a marking tends to be interreted in a similar way across a iece having two occurrences o the articular marking, in contrast to markings having more than two occurrences. This henomenon could be exlained more based on eatures such as the distance between the same markings, the length o the iece, or the number o dierent markings in the iece. Two cases exist where the P value1 is bolded (and red) and P value2 is not. The two cases are the two s in Mazurka O. 6 No. 1, and the three s in Mazurka O. 67 No. 3. No signiicant dierence was ound according to the one-way ANOVA (P > 0.05), but the grous indeed dier according to the two-way ANOVA (P < 0.05). This means that the grou means across recordings are similar, but the means diverge when considering the treatments o markings across recordings. Fig. 4 illustrates the resonses to the s in Mazurka O. 6 No. 1 and to the s in Mazurka O. 67 No. 3 in searate grahs. The x-axis indexes the recordings. The y-axis shows the loudness values. In each grah, horizontal lines show the mean loudness values or each instance o each marking. Dierent shaes indicate the dynamic at which each erormer realises each marking. Note in both grahs that the means or each erormer dier greatly across erormers, which exlains why P value2 is less than 0.05 or these two cases. 15

16 Mazurka # P value 1 P value 2 Mazurka # P value 1 P value 2 m M < < M < M M < M < < M < < M M < < M M < < M < < M M < < M < M < < M M < M M < < M M < < M M M M < M M < < M < M < < M M < M < < M < < M < < M M < M < M < < M < < M < < M < < M < < M M < M < < M < < M M < < M M < M < M < < M < < M < < M <10 10 <10 10 M < < M M < < M <10 10 <10 10 M < < M M < < M M < < M <10 10 M < < M < < M M <10 10 <10 10 M M <10 10 <10 10 M M <10 10 <10 10 M <10 10 M <10 10 <10 10 M < M <10 10 <10 10 M <10 10 <10 10 M <10 10 <10 10 M M <10 10 <10 10 M < < M <10 10 <10 10 M <10 10 <10 10 M M <10 10 <10 10 M M <10 10 <10 10 M M <10 10 <10 10 M < < M M < M <10 10 <10 10 M M <10 10 <10 10 M <10 10 M M M < < M M <10 10 <10 10 M M <10 10 <10 10 M <10 10 Table 4: Results or one-way (P value 1) and two way (P value 2) ANOVA tests or marking grous er Mazurka. P < 0.05 indicates the conclusion that the data means o the grous o values that corresond to a seciic dynamic marking dier. The symbol # indicates the number o the same markings that aear in the seciic Mazurka. 16

17 Loudness values o the two ortissimo markings in Mazurka O. 6 No mean(1) mean(2) Loudness values o the three Number orte o markings recordingin Mazurka O. 67 No mean(1) mean(2) mean(3) Number o recordings Figure 4: The case o the loudness values o grous o dynamic markings ( s in Mazurka O. 6 No. 1 to, and s in Mazurka O. 67 No. 3 bottom) or which the oneway ANOVA shows no signiicant dierence in the marking grou means (dotted lines), but the two-way ANOVA, which relects the diversity in the marking grou means er ianist, shows the oosite. In order to urther analyse the cases where the markings aeared in the score o the same Mazurka more than once and their treatments are ound to be dierent, we imlement the multile comarison test o means. Seciic cases are highlighted in Fig. 5. In each lot in Fig. 5, the conidence interval o an instance o a marking is shown as a horizontal line. The length o each line or each igure is the same because the conidence interval is comuted or the marking in that Mazurka. The vertical lines delineate author-selected highlight regions. We consider each lot in turn: Mazurka O. 6 No. 3, s: the markings between the vertical bars 4, 5, 6, and 7 are art o a reeated hrase with alternating s and s, as shown in Figure 6; as a result, their treatment is starkly dierent than that or the other s. Mazurka O. 24 No. 4, s: the irst and last marking are almost erectly aligned, giving the imression o an 17

18 Multile comarison o means Mazurka O. 6 No. 3 Mazurka O. 24 No. 4 Marking (a) Mazurka O. 63 No. 1 Marking (b) Mazurka O. 68 No. 2 Marking Marking (c) Mazurka O. 7 No (d) Mazurka O. 17 No Marking 2 3 Marking (e) () Figure 5: Multile comarison o the means or the grous o the same markings: (a) twelve s in Mazurka O. 6 No 3, (b) eleven s in Mazurka O. 24 No. 4, (c) our s in Mazurka O. 63 No. 1, (d) ive s in Mazurka O. 68 No. 2, (e) our s in Mazurka O. 7 No. 3, and () three s in Mazurka O. 17 No. 2. overall loudness stability, but the dynamics o this marking varies greatly in between. Mazurka O. 63 No. 1, s: the irst is signiicantly louder than the ones that ollow. Mazurka O. 68 No. 2, s: the irst is signiicantly soter than the ones that ollow. In both cases, excet or the outlying starting resonse, the means o the resonses to other instances o the marking are similar. Mazurka O. 7 No. 3, : the resonses to the markings are decreasing over time. Mazurka O. 17 No. 2, : the resonses to the markings are increasing over time. These 18

19 Figure 6: The reeated hrase in Mazurka O. 6 No. 3 where the markings 4, 5, 6 (reetition o 4 ), and 7 (reetition o 5 ) aear. two cases demonstrate two ossible resonses to sequences o the same markings. In summary, we have studied the ways in which dynamics evolve over time using one- and two-way ANOVA tests and multile comarison tests o means. We have shown that, most o the time, there exists a signiicant dierence between dynamic values or the same markings that aear in the score more than once. We rejected the hyothesis that the means are drawn rom the same oulation when analysing mean resonses to the same marking, as well as variations in individual recordings. Also, we have shown the kinds o atterns that emerge in resonses to sequences o the same dynamic markings across scores. 5 Clustering o dynamic change curves A question that ollows naturally is the degree o variation in dynamic behaviour across a single recording. In this section we resent an exeriment described in [22] where the MazurkaBL dataset is used. For this study, we create our data in a airwise manner by subtracting rom the dynamic value l b o marking b the dynamic value l (b 1) o the revious marking. The result is discretised as ollows: 2, x > 0.5 1, 0 < x 0.5 (x) = 0, x = 0, (6) 1, 0.5 x < 0 2, x < 0.5 where the inut x = l b l (b 1) or every marking osition b in each Mazurka. Then we ma each recording to a time series o discretised values o the unction. Each recording becomes a curve with discretised oints in score time. The urose o the rocedure above is to create meaningul clusters o curves so as to urther analyse the resulting shaes, and distinguish unusual behaviours. We use k-means to cluster the curves obtained. The number o clusters, k, is deined using the ga statistic [35]; we urther limit k to the range [3, 8] to ensure a reasonable number o recordings aears in each cluster, but also rovide the lexibility o detecting clusters that include unusual curves. 19

20 The result is a number o loudness behaviour clusters or each Mazurka. Clusters with the least number o recordings are labelled outliers. Then, we create a list o ianists that aear in these outlier clusters. Table 4 resents the to three outlier ianists who have each made recordings o more than thirty Mazurkas. Pianist (year o recording) # Outlier clusters # Recordings Ratio Cortot (1951) Poblocka (1999) Sztomka (1959) Table 5: Pianists having the highest roortion o recordings in outlier clusters. Magin is the only ianist whose recordings were the only element in a cluster or Mazurka O. 7 No. 1 and Mazurka O. 33 No. 3. Fig. 7 shows the loudness behaviour o Magin s recordings o these two Mazurkas, and how they dier rom the other recordings. Fig. 7 shows the centroid or each cluster, thus the discretised dynamic value may not be equal to any one outcome o the unction. Magin aears to dier rom the other recordings in interreting the last two transitions in Mazurka O. 7 No. 1, these are rom to and rom to, over the last three markings out o a total o thirteen markings. This shows that even though the values o Magin s cluster are airly close to the corresonding centroids o other clusters, the alternative interretation o the last marking was crucial in searating his recording rom others. In Mazurka O. 33 No. 3 Magin interrets all our markings in a contrarian ashion, thus resulting in the creation o a searate cluster or his unique recording. Next, we ocus on commonalities in unusual interretations. For this, we consider ianists whose recordings are in outlier clusters or every Mazurka, and we comute the number o times two ianists recordings are classiied in the same outlier cluster or all Mazurkas. The results are shown in Table 5, where we give the number o outlier clusters containing the seciic ianist airs. We ocus on airs that co-occur in outlier clusters or more than twenty Mazurkas. Pianist Fliere Milkina Ezaki Barbosa Chiu Smith Rubinstein Kushner Czerny-Steanska Table 6: Pianists whose recordings co-occur in more than twenty outlier clusters. Table 6 shows higher degrees o similarity in erormed loudness between ianist airs Chiu Milkina, Chiu Barbosa, Smith Fliere, Rubinstein Milkina, Kushner Ezaki, and Czerny- Steanska Barbosa. Thus, although certain ianists are more oten in outlier clusters, the ways in which they dier in their loudness interretations oten ollow shared atterns. Further work remains to trace their genealogy to determine teacher inluences in their laying styles. 20

21 Discretized dynamic value Cluster 1 (3) Cluster 2 (6) Cluster 3 (12) Cluster 4 (8) Cluster 5 (1) Cluster 6 (8) Cluster 7 (3) Mazurka O. 7 No (,) (,) (,) (,) (,) (,) (,) (,) (,) (,) (,) (,) Marking air 2 Mazurka O. 33 No. 3 Discretized dynamic value Cluster 1 (16) Cluster 2 (1) Cluster 3 (6) (,) (,) (,) Marking air Figure 7: Loudness transition clusters ound or Mazurka O. 7 No. 1 (to) and O. 33 No. 3 (bottom); in each case, the cluster comrising o only Magin s recording is shown in solid bold lines, the cluster having the highest number o recordings is shown as dotted bold lines. 6 Conclusions In this article we have investigated the relationshi between loudness levels and dynamic markings in the score. The statistical analysis resented rejects hyotheses such as a dynamic marking is erormed in the same way when it aears in the score more than once and each erormance ollows the dynamic structure o the iece as indicated by the score markings. Our indings underin the statement that while modern interretations are more aithul to the score [31,.4],the maniested sound or the same eect can be distinct and vary greatly between erormers, between erormances, and within the same erormance. In addition, our indings reveal that one simle rule or each dynamic marking cannot suice to deine its ossible dynamic levels; in act, the realised dynamics are constrained by 21

22 many other actors, the most highlighted in this article o which are: the current, revious, and next dynamic markings; the distance rom the current marking to the revious and next markings; the nearest non-dynamic marking annotated between the revious and current or next dynamic marking, such as crescendo; and, any qualiying annotation aearing simultaneously with the current dynamic marking, such as dolcissimo. Whole-iece atterns in dynamics are discovered through studying dierent ianists interretation o the markings in the score. We were able to identiy outlier loudness interretation behaviours, and joint behaviours. Although certain ianists exhibit outlier behaviours across a large number o Mazurkas, when these ianists outlier behaviours co-occur with other ianists, they tend to do so airly consistently across the Mazurka reertoire. 7 Discussion The actors that have been created through this study have layed an imortant role in [24]; they constitute the list o eatures that have been extracted in order to investigate the bidirectional maing between dynamic markings in the score and erormed loudness. The study alied machine-learning techniques to the rediction o loudness levels corresonding to dynamic markings, and to the classiication o dynamic markings given loudness values. The results show that loudness values and markings can be redicted relatively well when trained on dierent recordings o the same iece, but ail dismally when trained on the ianist s recordings o other ieces, demonstrating that score eatures may trum individual style when modelling loudness choices. The evidence suggested that all the eatures chosen or the rediction and classiication tasks current/revious/next dynamic markings, distance between markings, and roximity o dynamic-related and non-dynamic markings were relevant. Furthermore, analysis o the results reveal the orms (such as the return o the theme) and structures (such as dynamic marking reetitions) inluence the redictability o loudness levels and dynamic markings. Future stes include the exansion o the current list o actors, by incororating new arameters related to additional score notation. One arameter could be the number o notes that are layed simultaneously at a marking osition as this may aect the resulting loudness. Also the extraction o structural arameters, such as the marking osition as it relates to ositions within hrase could warrant urther investigation; there are many ways to interret a music iece utilizing both comositional and exressive arameters, with resect to structural asects such as hrasing [34] and itch [14]. With regard to temo changes; [36] examined temo-loudness interactions at seciic score markings over a subset o the MazurkaBL dataset, and investigated how including inormation on one arameter imacted rediction o the other. The authors considered score markings indicating loudness or temo change, and the model included score, temo, and loudness-related eatures. When considering recordings o the same Mazurka, exeriments showed that considering loudness-related eatures did not imrove rediction o temo change. However, adding temo-related eatures did result in marginal imrovement in redicting loudness change. Other uture directions include the exloration o new ways to cature the loudness level corresonding to a marking using dierent loudness detection techniques. 22

23 It is imortant to remain conscious o the limitations o score based analyses: deviations rom the score can occur in erormance, such as laying notes in an octave dierent rom that written or alying rubato where not indicated. More general, working rom scorebased analysis to recording may be considered as declaring o limits all those asects o erormance that cannot be directly related to notational categories, eliminating the unnotatable asects o rhetoric, ersuasion, and exressive eects having little or nothing to do with theoretical structure [8,. 233]. Other deviations rom the score occur when erormers reer an aberrant interretation style which deviates rom the score. From the standoint o the researcher, it can be unclear whether the exressive deviations measured are due to deliberate exressive strategies, music structure, motor noise, imrecision o the erormer, or even measurement errors [25], or even well-known scores that aear to have widely-recognised meanings are changing all the time, not simly as general erormance style changes but in their characterisation, leading to a change in their erceived nature. [26] It is not clear what an analysis o the dynamics o a erormance reresents when based on an audio recording. Do the dynamics result rom the erormer s intention or the audio engineer s ercetion? The mastering o the sound during the roduction rocess can indeed aect the result o what we hear, which could be dierent rom what was actually layed. The tye o instrument and the room acoustics may also aect the end result. In our case, the data is taken rom studio or roessional live recordings and the musician s consent to the inal result is assumed. The above comments oint to the inherent comlexity o the subject matter under investigation; as such, the results reorted should be seen as roviding a sound basis or a more sohisticated aroach to understanding the relativity o exression o dynamic markings. Acknowledgements This research has been unded in art by a Queen Mary University o London Princial s studentshi. The authors would like to thank Jordan Smith, Ph.D., or advice on statistical tools used or this research. Reerences [1] Eleanor Bailie. Choin: A Graded Practical Guide. Pianist s reertoire. Kahn & Averill, isbn: url: htts://books.google.co.uk/books?id= eoobqgaacaaj. [2] Jacob Beck and William A. Shaw. Ratio-estimations o loudness intervals. In: Journal o the acoustical society o America 80 (1967), [3] Alonso Benetti Jr. Exressivity and musical erormance: Practice strategies or ianists. In: Perormance Studies Network International Conerence, Cambridge url: htt:// d. 23

24 [4] Stean Benus, Agustín Gravano, and Julia Hirschberg. Prosody, emotions, and... whatever. In: Interseech, Antwer url: htt://www1.cs.columbia.edu/nl/aers/ 2007/benus_al_07a.d. [5] Erica Bisesi and Richard Parncutt. Second Vienna Talk. In: Proceedings o the Second Vienna Talk, University o Perorming Arts Vienna. 2010, [6] R. A. W. Bladon and Björn Lindblom. Modeling the judgment o vowel quality dierences. In: The Journal o the Acoustical Society o America 69.5 (1981), [7] Nicholas Cook. Music: A Very Short Introduction. Very Short Introductions. OUP Oxord, isbn: url: htts://books.google.com.au/books? id=dl2gvbikvec. [8] Nicholas Cook et al. The Cambridge comanion to recorded music. Cambridge University Press, [9] Michael Scott Cuthbert and Christoher Ariza. music21: A Toolkit or Comuter- Aided Musicology and Symbolic Music Data. In: Proceedings o the 9th International Conerence on Music Inormation Retrieval. 2010, [10] Alred Einstein. Music in the romantic era: a history o musical thought in the 19th century. New York : W.W. Norton, [11] Sebastian Ewert, Meinard Müller, and Peter Grosche. High resolution audio synchronization using chroma onset eatures. In: thirty-ourth IEEE International Conerence on Acoustics, Seech, and Signal Processing (ICASSP). Taiei, Taiwan, 2009, [12] Dorottya Fabian, Renee Timmers, and Emery Schubert. Exressiveness in music erormance: Emirical aroaches across styles and cultures. Oxord University Press, [13] Sebastian Flossmann et al. The Magalo roject: An interim reort. In: Journal o New Music Research 39.4 (2010), [14] Anders Friberg, Roberto Bresin, and Johan Sundberg. Overview o the KTH rule system or musical erormance. In: Advances in Cognitive Psychology (2006), [15] John M Geringer. Loudness estimations o noise, synthesizer, and music excerts by musicians and nonmusicians. In: Psychomusicology: A Journal o Research in Music Cognition 12.1 (1993),. 22. [16] Werner Goebl. Melody lead in iano erormance: Exressive device or artiact? In: The Journal o the Acoustical Society o America (2001), [17] Maarten Grachten and Florian Krebs. An Assessment o Learned Score Features or Modeling Exressive Dynamics in Music. In: IEEE Transactions on Multimedia 16.5 (Aug. 2014), issn:

25 [18] Maarten Grachten and Gerhard Widmer. Linear Basis Models or Prediction and Analysis o Musical Exression. In: Journal o New Music Research 41.4 (2012), erint: htt://dx.doi.org/ / url: htt://dx.doi.org/ / [19] Maarten Grachten et al. Towards comuter-assisted understanding o dynamics in symhonic music. In: IEEE Multimedia: Secial Issue on Multimedia Technologies or Enriched Music Perormance, Production and Consumtion 24.1 (2017), [20] Agustín Gravano et al. On the role o context and rosody in the interretation o okay. In: Proceedings o the 45th Annual Meeting o the Association o Comutational Linguistic (ACL). 2007, [21] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew. MazurkaBL: Score-aligned loudness, beat, exressive markings data or 2000 Choin Mazurka recordings. In: (to aear) [22] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew. Outliers in erormed loudness transitions: an analysis o Choin Mazurka recordings. In: Proceedings o the 14th International Conerence or Music Percetion and Cognition (ICMPC). San Fransisco, Caliornia, USA, 2016, [23] Katerina Kosta, Oscar F. Bandtlow, and Elaine Chew. Practical Imlications o Dynamic Markings in the Score: Is Piano Always Piano? In: Audio Engineering Society (AES) 53rd International Conerence on Semantic Audio. London, UK, [24] Katerina Kosta et al. Maing between dynamic markings and erormed loudness: A machine learning aroach. In: Journal o Mathematics and Music 10.2 (2016), erint: htt://dx.doi.org/ / url: htt://dx.doi.org/ / [25] Jörg Langner and Werner Goebl. Visualizing Exressive Perormance in Temo- Loudness Sace. In: Comuter Music Journal 27.4 (Dec. 2003), issn: url: htt://dx.doi.org/ / [26] Daniel Leech-Wilkinson. Comositions, scores, erormances, meanings. In: Music Theory Online 18.1 (2012), url: htt://mtosmt.org/issues/mto /mto leech-wilkinson.h. [27] MazurkaBL dataset. htts://github.com/katkost/mazurkabl. Accessed: [28] Regina Nuzzo. Scientiic method: Statistical errors. In: Nature (2014), url: htt : / / www. nature. com / news / scientiic - method - statistical - errors [29] Caroline Palmer and Sean Hutchins. What Is Musical Prosody? In: vol. 46. Psychology o Learning and Motivation. Academic Press, 2006, url: htt: // [30] E. Pamalk, A. Rauber, and D. Merkl. Content-based Organization and Visualization o Music Archives. In: Proceedings o the ACM Multimedia. Juan les Pins, France: ACM, Dec. 2002,

26 [31] John Rink. Musical Perormance: A Guide to Understanding. Cambridge University Press, isbn: url: htts://books.google.co.uk/books?id= xayxre-5ztic. [32] María Ros et al. Transcribing Debussy s Syrinx dynamics through Linguistic Descrition: The MUDELD algorithm. In: Fuzzy Sets and Systems 285 (2016), url: htt://dx.doi.org/ /j.ss [33] Manred R Schroeder, Bishnu S Atal, and JL Hall. Otimizing digital seech coders by exloiting masking roerties o the human ear. In: The Journal o the Acoustical Society o America 66.6 (1979), [34] J. Sundberg. Gluing tones: grouing in music comosition, erormance and listening. Kungl. Musikaliska akademiens skritserie. Royal Swedish Academy, isbn: url: htts://books.google.co.uk/books?id=em45aqaaiaaj. [35] Robert Tibshirani, Guenther Walther, and Trevor Hastie. Estimating the number o clusters in a data set via the ga statistic. In: Journal o the Royal Statistical Society: Series B (Statistical Methodology) 63.2 (2001), [36] Carlos Vaquero, Ivan Titov, and Henkjan Honing. What score markings can say o the synergy between exressive timing and loudness. In: roceedings o the Euroean Society or Cognitive Sciences O Music (to aear in July). [37] Gerhard Widmer and Werner Goebl. Comutational Models o Exressive Music Perormance: The State o the Art. In: Journal o New Music Research 33.3 (2004),

27 8 Aendix The grahs below show the dynamics at all beats, not only the ones at the dynamic markings. The variety o curves in the background shows the diversity o dynamic resonses rom the ianists at the same score beats (x axis). The grahs also show how the resonses vary or each marking. Wherever a symbol in a air o markings is resent, a boxlot indicates the sread o loudness values at that osition. The eye in the middle o each boxlot marks the median, and the to and bottom edges indicate the 75 th and 25 th ercentiles; the whiskers extend to the most extreme data oints excluding the outliers. Mazurka cases where E() > E() marking air - is only one with negative τ value. There are two cases, Mazurka O. 30 No. 3 in D major, and Mazurka O. 68 No. 3 in F major, where there is only one negative τ value, which occurs at the marking air -, meaning that a signiicant number o individual recordings have E() > E(). Fig. 8 shows the dynamic values or all recordings o these Mazurkas in score time, and the distribution o the ianists resonses to the markings and in articular. In Mazurka O. 68 No. 3, we notice that the average dynamic o the irst is louder than that o any other marking, including that o. One reason or this much higher irst may be its location on the irst beat o the iece, the result o giving extra emhasis to the beginning. Also, by observing the changes in temo, we notice that the erormers tend to lay slower at the locations o the and the second. In Fig. 9, the irst two bars show the score area where the a marking is located, while the last two bars show another marking; it is noticeable that although both locations belong to the start o eectively the same two-bar hrase, the slight dierence in the atterns makes ianists use dierent ingerings, which may add to the diverse resonse. Another oint o note is that beore the second marking there is a twelve-bar new hrase at the key o B major (not shown). In Mazurka O. 30 No. 3 the average loudness value o the three s is less than that or the eleven s in a signiicant number o recordings. The osition o the s in the score oer an exlanation or this result. The three markings belong to the same reeated hrase which is shown in Fig. 10 and they are located between two markings. Their duration is one score bar. In every recording, the average is signiicantly soter than the average (τ = 1). This means that there is indeed a loudness change during the score sequence - -. The average s are louder, erhas because the did not have to be so much louder than the s to make a large contrast. The resonse to the s nearest the - - trios are louder on average than the s s, as shown in Fig. 10. Reasons or this include the crescendo (<) marking right ater each, an instruction to increase in loudness, and the act that almost all ianists aly a crescendo right ater the second in this excert, which amliies the dynamic level o the that ollows. 27

28 1 Mazurka O. 68 No Markings Figure 8: The Mazurka cases where only the - air has negative τ value. The dynamic values o the markings belonging to the air in Mazurka O. 30 No. 3 (to), and Mazurka O. 68 No. 3 (bottom) are resented as box lots. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. Figure 9: The reeated hrase where the aears in Mazurka O. 68 No. 3 (let art: score beats 1 2, right art: score beats ) 28

29 Figure 10: The reeated hrase where the aears in Mazurka O. 30 No. 3 and its relation to the neighbouring markings, and (score beats 25 54, , and ) Mazurka cases where E() > E() marking air - is only one with negative τ value. There are three cases, Mazurka O. 33 No. 4 in B minor, Mazurka O. 67 No. 3 in C major, and Mazurka O. 68 No. 2 in A minor, where there is only one negative τ value which is at the marking air -, meaning that a signiicant number o individual recordings have E() > E(). Fig. 11 shows the dynamic values or all recordings o these Mazurkas in score time, and the distribution o the ianists resonses to the marking airs - in articular. In Mazurka O. 33 No. 4, there is one marking layed louder than the two markings on average. One exlanation is the loudness rogression throughout the duration o the (in the ga between the and markings), as it can be observed in the to lot o Fig. 11. Many bars searate the and the ensuing, as shown in Fig. 12, the irst hal o these intermediate bars are louder than the second hal, resulting in a signiicant loudness dro that leads into the marking. In Mazurka O. 67 No. 3, there are our s in a row which are louder on average than the three s. More seciically, the three s are located at the beginning o the same hrase which is reeated, a act that likely exlains the very similar loudness level at the location o all the s. As Fig. 13 shows, every is receded by a sustained note in s, which might be the reason or the higher loudness level. In Mazurka O. 68 No. 2, there is a middle art, where there are our closely clustered hums, as they can be observed rom the background curves in Fig. 11, which corresond to a reeated section, where there are and markings. In this middle art, the s are indeed soter than the s on average. However, these s are on average louder than the average o the remaining s. 29

30 1 Mazurka O. 33 No Markings Mazurka O. 67 No Markings Mazurka O. 68 No m m m Markings m m m m Figure 11: The Mazurka cases where only the - air has negative τ value. Box lots o the dynamic values o the markings belonging to the air in Mazurkas O. 33 No. 4, O. 67 No. 3, and O. 68 No. 2 are resented. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. 30

31 Figure 12: The osition o the marking in Mazurka O. 33 No. 4. The resonse o the recordings at this osition is higher than the resonse at the osition o the marking on average. Figure 13: The markings in Mazurka O. 67 No. 3. The resonse o the recordings at these ositions is higher than the resonse at the ositions o the markings on average. Mazurka cases where E() > E(m ) marking air -m is only one with negative τ value. There are two cases, Mazurka O. 50 No. 1 in B minor, and Mazurka O. 56 No. 3 in C major, where there is only one negative τ value which is at the marking air -m, meaning that a signiicant number o individual recordings have E() > E(m). Fig. 14 shows the dynamic values or all recordings o this Mazurka in score time, and the distribution o the ianists resonses to the marking airs -m in articular. In the case o Mazurka O. 50 No. 1 the loudness values at the single m are lower than the ones at the s on average. Observe in Fig. 14 that the marking that recedes the m is shown to be louder, but there is a loudness dro in the intervening measures, meaning that there is indeed an increase in loudness at the m in most recordings. Fig. 15 shows the score 31

32 m 1 Mazurka O. 50 No Markings 1 Mazurka O. 56 No m Markings Figure 14: The Mazurka cases where only the -m air has negative τ value. The dynamic values o the markings belonging to the air in Mazurka O. 50 No. 1 (to), and Mazurka O. 56 No. 3 (bottom) are resented as box lots. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. osition or both markings. The loudness value decreases as the hrase closes beore the m, and is locally ascending around the m, but this change is not catured when considering only loudness at the markings. In the case o Mazurka O. 56 No. 3, the single m marking is located at the beginning o the iece and it is reorted as being less loud than the average o the markings. In all but one case, the marking receding a is, and in three o these cases, the is in the 32

33 Figure 15: The m marking in Mazurka O. 50 No. 1 and its relation with the receded marking. bar immediately beore the. When the markings are very close, there is too little time to realise a and the eect o the dro in loudness can only be detected in later bars. The case o Mazurka O. 41 No. 2: E() > E() and E(m) > E(). In Mazurka O. 41 No. 2, E() > E(), and E(m) > E(). For the -, and m - airs in this Mazurka, the τ value is negative, meaning that a signiicant number o individual recordings have E() > E(), and E(m) > E(). Fig. 16 shows the dynamic values or all recordings o this Mazurka in score time, and the distribution o the loudness values at the markings, m, and in articular. 1 Mazurka O. 41 No m Markings Figure 16: Box lots o the dynamic values o the markings belonging to the airs -, and m - in Mazurka O. 41 No. 2. Positions o other markings s S aear in x- ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. Note that the irst marking is generally layed soter than the second one, and this aects the average level o the resonse to this seciic marking. In order to exlore this behaviour, 33

34 we resent in Fig. 17 the location o the two s as they aear in the score. The irst one is receded by the text indicator (dim.), which lowers the threshold or change ercetion at the. This is ollowed by a short Crescendo, which allows the to exand in loudness ater the marking; in this case, the ianist must allow room or this exansion. The section concludes with the text indicator (dim.). Next, we examine the next two observations ollowing the. The marking, which is relatively loud, is located at the beginning o the score, where ianists are more likely to lace more emhasis on the oening notes. Also, the m marking, which can be seen at the bottom art o Fig. 17, is inside arentheses, which suggests reedom on how it could be interreted. Figure 17: The location o the irst marking (to), and the location o the second marking in Mazurka O. 41 No. 2 (bottom), score-beats 40 57, and , resectively. 34

35 The case o Mazurka O. 50 No. 3: E() > E(), E()>E(), and E()>E(). In Mazurka O. 50 No. 3, an irregularity that is related to the interretation o the marking is observed. More seciically, the single marking, located at the enultimate score measure, is reorted as lower in loudness level than the average loudness o the s, s, and s, resectively. The counter-intuitive loudness transitions (negative τ values) or the airs,, and is caused by the act that the single at the end is layed extraordinarily sot. Fig. 18 shows this inding, as well as the score rogression or extremely well-behaved (τ equals 1) loudness airs, and, and the relatively small agreement (τ value equals to 0.134) or the air. 1 Mazurka O. 50 No Markings Figure 18: Box lots o the dynamic values o the markings belonging to the airs (, ), (, ), and (, ) in Mazurka O. 50 No. 3. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. 35

36 The case o Mazurka O. 67 No. 1: E() > E(m), and E() > E(m). In Mazurka O. 67 No. 1, the marking airs -m, and -m have negative τ values, meaning that a signiicant number o individual recordings have E() > E(m), and E() > E(m). Fig. 19 shows the distribution o the loudness levels throughout the recordings at the ositions where the seciic markings aear in the score. As a side note we should highlight that this is the only Mazurka rom the ones we test which includes all the markings S at least once. 1 Mazurka O. 67 No m Markings Figure 19: Box lots o the dynamic values o the markings belonging to the airs (, m ) and (, m ) in Mazurka O. 67 No. 1. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. In order to exlore the reason why the markings seem louder in average than the m marking, we could ocus on the act that all s, excet rom the third one, are ollowed by a Crescendo marking directly ater the beat in which they aear. Consider the irst marking, which is ollowed by a Crescendo, and receded by a marking one-score beat away. The markings are each receded by a marking two score-beats away, which may aect the overall higher resonse to the s, although the balance o E() < E() is ket in most o the recordings. The case o Mazurka O. 67 No. 2: E()>E(), and E() > E(m). In Mazurka O. 67 No. 2, the marking airs -, and -m have negative τ values, meaning that a signiicant number o individual recordings have E() > E(m), and E() > E(m). Fig. 20 shows that the average resonse to is louder than the resonse to m, and to in most o the recordings. Both s location in a reeated hrase in the score is shown in Fig. 20. Observing how the loudness changes have been reorted throughout the recordings, we notice that the second in the reeated hrase is soter than the although there is a crescendo marking in the revious measure. This may be related to the act that in most recordings the ianists choose to emhasise the hrase closing, which is where the seciic marking is located. Then the m marking that ollows is louder than the, but it 36

37 ails to suersede the robust loudness values o the s. 1 Mazurka O. 67 No Markings Figure 20: Box lots o the dynamic values o the markings belonging to the air and the air m in Mazurka O. 67 No. 2. Positions o other markings s S aear in x-ticks as in score sequence. At the background the curves reresent the score beat dynamic values er recording. m Figure 21: The reeated hrase in Mazurka O. 67 No. 2 which includes the marking, and the two markings that ollow u. 37