THE EXPRESSIVE NOTATION PACKAGE

Transcription

1 Mika Kuuskankare THE EXPRESSIVE NOTATION PACKAGE A final report in the Development Study Programme submitted in partial fulfillment of the requirements for the degree of Doctor of Music DocMus Department of Doctoral Studies in Musical Performance and Research Sibelius Academy Helsinki, Finland 2006

2 2

3 The Development Study Programme Portfolio This report is part of the portfolio required for the Development Study Programme for the degree of Doctor of Music. It consists of an introductory essay and seven peerreviewed articles. The portfolio comprises this report, a cross-platform music notation program the Expressive Notation Package, or ENP conceived by the author, a Reference Manual, and an interactive Tutorial both provided as a part of the ENP program. ENP is primarily intended for composers and music analysts. It is meant to be used as a notational front-end in a visual programming language called PWGL. PWGL, in turn, is a cross-platform general-purpose visual programming language that the author has co-developed in collaboration with Mikael Laurson (the lead developer) and Vesa Norilo. ENP is distributed as a part of PWGL and can be downloaded at the following address: 3

4 4

5 Preface I would like to express my gratitude to the following people. The pre-examiners of my portfolio, Professor Roger B. Dannenberg and Dr. Baptiste Barrière, for their invaluable insights and comments. The members of my project advisory group: Professor Marcus Castrén, Dr. Mikael Laurson, and Professor Matti Karjalainen. My supervisor, Professor Marcus Castrén, for convincing me to become a doctoral student. Dr. Mikael Laurson for essentially putting all this in motion about 15 years ago by introducing me to Lisp. I would also like to thank him for his support and friendship over the years. Jean- Dr. Annikka Konttori-Gustafsson, the wise and lovely leader of the DocMus Department, for her gracious attitude towards my work. My dear friend, the saxophone artist Olli-Pekka Tuomisalo, for his extraordinarily uncomplicated friendship, and for the shared enthusiasm for high frequencies. The composer Kimmo Kuitunen for sharing his insightful knowledge and understanding of music, and also for his persistence in learning about ENP. The Boston Promenade Big Band for providing a good home for the out-of-tune-toohigh-too-loud notes that I can generate with my trumpet. The wave-guide model is on the way... The Sibelius Academy Innovation Centre for the four-year postgraduate funding that enabled me to carry out my research. Joan Nordlund for checking the language. Last, but definitely not least, I wish to thank my mother, who has supported me in many different ways over the years. 5

6 6

7 Contents List of Publications 9 I Introductory Essay 11 1 Introduction Background Overview of ENP ENP Applications The ENP Project The Scope of this Report The Contents of this Report The Structure of this Report A Note on Terminology Supplementary Examples Notation Modern Notation Orchestral Notation Tape Notation Music Analytical Notation ENP-Expressions Lyrics Dynamic Expressions Expressions Containing Break-point Functions Instrument-sensitive Expressions Auto-canceled Expressions ENP-Script Input Shortcut Inputting Rhythm

8 8 Contents Score Manipulation Analysis and Visualization Conclusions Summary Future Directions Short-term Enhancements New Avenues of Research A Summary of the Publications 61 References 63 II Publications 71 Publication [P1] 77 Publication [P2] 91 Publication [P3] 97 Publication [P4] 102 Publication [P5] 113 Publication [P6] 121 Publication [P7] 126

9 List of Publications The report consists of an introductory essay and the following seven articles referred to as [P1] [P7]. [P2] [P7] are verbatim copies of the published articles, [P1] is attached as a revised manuscript. [P1] [P2] [P3] [P4] [P5] [P6] M. Kuuskankare and M. Laurson. Expressive Notation Package. Computer Music Journal, 30(4):67 79, Winter (in press; attached as a manuscript). M. Kuuskankare and M. Laurson. Annotating Musical Scores in ENP. In International Symposium on Music Information Retrieval, London, UK, M. Kuuskankare and M. Laurson. Intelligent Scripting in ENP using PWConstraints. In Proceedings of International Computer Music Conference, pages , Miami, USA, M. Laurson and M. Kuuskankare. From RTM-notation to ENP-score-notation. In Journées d Informatique Musicale, Montbéliard, France, M. Kuuskankare and M. Laurson. Recent Developments in ENP-score-notation. In Sound and Music Computing 04, Paris, France, M. Kuuskankare and M. Laurson. ENP Expression Designer - a Visual Tool for Creating User Definable Expressions. In International Computer Music Conference, pages , Barcelona, Spain, [P7] M. Kuuskankare and M. Laurson. ENP Note Head Designer. In Journées d Informatique Musicale, Paris, France,

10 10

11 Part I Introductory Essay 11

12

13 Chapter 1 Introduction The focus of attention in this report is a computer-based music notation application, the Expressive Notation Package or ENP, conceived by the author. ENP is among the various music-related software packages developed at the Sibelius Academy during the past 20 years. Since its inception it has been used extensively in several research projects. 1.1 Background Classical music is primarily a written tradition. Throughout the centuries musical compositions have been preserved in music notation, as opposed to being transmitted orally. At the present time, common music notation is the lingua franca of both music education and professional musicians. Regardless of its apparent success, it has also been subject to criticism, and many proposals have been made to modernize it. Yet, it is still the most complete and widespread method that we know of for notating the complex and interrelated properties of a musical sound: pitch, intensity, time, timbre, and pace. Common music notation is also an invaluable tool in the field of computer music. The history of computer-aided music notation begun as early as in the 1960s. To- 13

14 14 1. Introduction wards the end of the decade a few computer-based music notation programs started to emerge, and arguably one of the earliest systems was developed at the National Research Council in Ottawa, Canada [52]. In the 1970s several text-based programs were introduced, among others, Leland Smith s early work MS [52] and Donald Byrd s SMUT [11]. Among the first interactive editors were Scriva [10] and Mockingbird [31]. The first commercial notation programs (including, among others, Score [54] program by Leland Smith) were also introduced in the 1970s but it was not until the 1980s that these programs were made available to the general public. [P1] gives a capsule list of some of the current applications. Furthermore, a detailed history of computer-based music notation can be found in [52]. According to Byrd: [...] fully automatic high-quality music notation is not merely nontrivial but in general impossible without human-level intelligence.[11] The complexity of music notation makes it a difficult programming task. Creating a comprehensive computer-based music notation application requires addressing several challenging issues, including, but not limited to: 1. computer representation of music notation [11, 16, 15, 19, 57, 28, 27, 7, 22, 37]; 2. visualization of musical information [32, 58]; 3. automatic spacing algorithms [25, 8, 51]; 4. data formats, structures and languages [11, 24, 13, 46, 29]; 5. scripting languages (e.g., ManuScript of Sibelius [53]); and 6. user-interface [14, 12, 6, 48]. Advanced computer-based music notation has several interesting applications. It can be used to produce exceptionally sophisticated results, for example, in computer

15 1.2 Overview of ENP 15 based performance of music. Although musical performance in this regard is becoming increasingly life-like there is still room for improvement in terms of controllability and naturalness. The interaction between computer-based music notation and sound source control which allows for the production of convincing renditions of musical compositions remains among the most interesting applications. Computer-aided composition and music analysis present challenges in terms of visualization, representation, and accessibility. Algorithmic control requires a versatile representation of the complex structural and notational information behind the visual front-end of a musical score. Computer-assisted instruction, in turn, requires a high level of interactivity and built-in musical knowledge. This list of applications, although not exhaustive, already introduces a number of challenges related to their potential implementation. 1.2 Overview of ENP ENP [35] is a music notation program that is written in Lisp, CLOS [55], and OpenGL [59]. It uses an implementation of ANSI Common Lisp by LispWorks (LW, LW is source code compatible across Windows, Linux, Mac OS X, and some UNIX platforms. ENP has been developed in order to meet the requirements of computer-aided composition, music analysis and synthesis control. It is intended to represent Western musical notation, currently excluding that written before the 17th century. There is strong emphasis on the applications of new music and contemporary notation (for examples see [9] and [33]). ENP is not a full-featured music typesetting program but it is designed to produce automatic, reasonable musical typesetting according to common practices (See [49] and [56]). The graphics part of ENP is programmed using OpenGL a cross-platform API for developing portable, interactive 2D and 3D graphics applications. It features doublebuffering, anti-aliasing and hardware acceleration, for example, and is an inexpensive

16 16 1. Introduction way of creating state-of-the-art graphical output and user interaction. One of its most important features is its ability to interact with arbitrary complex graphical objects. This mechanism called picking is a powerful tool when programming graphical user interfaces. ENP has an object-oriented graphical user interface (GUI) that is based on direct manipulation. According to Cooper [14], direct manipulation can be broken down into three basic principles: (1) visual representation of the objects, (2) the use of dragging instead of text entry or menu selection, and (3) feedback on the impact of the manipulation. Object-oriented UI, in turn, is an interface paradigm where the user first selects an object and then chooses an action from a context menu or via direct manipulation. The underlying idea behind ENP GUI is to allow the user to manipulate the information contained in a digital musical score as straightforwardly as possible. As a general guideline, any musical object of any complexity can be edited directly in the score. All editing operations also provide synchronized visual feedback for the user. ENP is primarily intended to be used as a notational front-end in a cross-platform general-purpose visual programming language called PWGL [41], which has a strong emphasis on music- and sound-related applications. It is a combination of several software packages built on top of Common Lisp and CLOS [55] its components including a rule-based programming language called PWGLConstraints [38], and the PWGLSynth [40] sound synthesis program. ENP is an important bridge between the different software components inside PWGL in that it allows the packages to inter-operate by using its representation of musical objects. It is closely integrated with PWGLSynth and enables PWGLConstraints to be used to generate musical material. It can also be used in PWGL to view and edit musical material in many different ways, and to algorithmically generate and process musical objects.

17 1.3 ENP Applications ENP Applications According to Glendon Diener [21], music notation programs have three basic types of use: compositional, archival, and analytic. In its current state ENP can be used as a compositional and an analytical tool, and also as an ordinary music notation program (archival use). Furthermore, a fourth category, algorithmic, is being added. The current applications of ENP could thus be loosely categorized as follows: 1. Archival (a) to represent complex music notation including special facilities for 20th century music (b) to manipulate a score through a mouse-driven GUI using direct editing (c) to produce high quality paper output using PostScript 2. Compositional (a) to generate musical material with the help of a rule-based language (b) to add, remove, and modify information (pitch, rhythm, markings) using an intelligent scripting language 3. Analytic (a) to annotate a score, either manually or automatically, to display music theoretical analysis information, e.g., motives, harmonic progressions, Schenkerian graphs, or pitch-class set theoretical information (b) to add visible or invisible notes to musical objects (c) to dynamically inspect and visualize data contained by the score 4. Algorithmic (a) to algorithmically generate and manipulate musical structures, scores, parts, chords, and notes, for example (b) to use a score as a user interface component

18 18 1. Introduction (c) to visually create custom notational attributes that are immediately usable without recompiling and (d) to textually describe a score and the individual notational objects. ENP has already been used as a tool in several different projects to produce control information for various virtual instrument models [39]. One of its unique aspects is that control information can be drawn directly on the score. Figure 1.1 shows visual control information specialized for a virtual guitar instrument model. An elaborate interpretation can be generated with the help of both standard and non-standard expressions. Here, the standard expressions include some idiomatic guitar markings, such as string numbers and left-hand slurs. The non-standard expressions, in turn, include a tempo function (marked as /time ), different kinds of sustains ( s and gs ), and a macro-note expression ( vlarp ) [44]. The playback information generated by the macro-note expressions can be seen in the lower staff written as stemless notes. Figure 1.1: A score written for a virtual guitar containing both standard and non-standard expressions. There are also some macro-note expressions that create complex arpeggio gestures (see the stemless notes in the lower staff). The score is a free adaptation of a guitar piece by Heitor Villa-Lobos (12 Etudes: No.1 Allegro Non Troppo). The composer Kimmo Kuitunen used ENP to visualize the ideas behind large-scale durations in his composition Hodoi tô Ergô [34]. He used it in another piece, Trio for

19 1.3 ENP Applications 19 Clarinet, Cello and Piano, to visualize its structural organization (see Figure 1.2). Here, the textual annotations above the notes denote different kinds of texture classes (SOL for solo, HOQ for Hoquetus, etc.). Furthermore, colors and colored shapes are used to indicate similar texture classes. 1 The durations of the notes, in turn, indicate the lifespan of the given texture class in 16 th notes (the number of required 16 th notes is also marked above the texture-class annotation). Figure 1.2: One potential view of the structural organization of a piece, demonstrating the combination of rhythmic information, textual annotations, colors, and shapes as a visualization aid (Kimmo Kuitunen: a visualization of the structure of his Trio for Clarinet, Cello and Piano. Here, the structure of roughly the first half of the first movement is shown). One interesting application of ENP is in computer-assisted music analysis and the visualization of the analysis information. Currently, there are several sophisticated music analysis applications available (e.g., HumDrum, [30]; athenacl, [4]) that use text-based representation as a starting point. This kind of approach is usually suitable 1 The colors are visible only in the electronic version of this report.

20 20 1. Introduction for analytical questions that are statistical in nature. However, it makes it more difficult to verify the result because the analysis is given as a separate textual document without the corresponding notational counterpart. ENP, in contrast, makes it possible to use a set of specialized graphical devices to visualize the analysis information directly in the score, and to automatize a large part of the analysis process. Figure 1.3 gives the realization of the first five texture classes that are shown in Figure 1.2. The correlation between the abstract structural outline and the final composition can now be seen more clearly. The texture classes are brought out by a line enclosing the corresponding musical material. To emphasize the association with the texture classes found in Figure 1.2, the original colors and names are used. Publications [P1] and [P2] give numerous examples of music analytical applications of ENP.

21 1.3 ENP Applications 21 Figure 1.3: The structural plan presented in Figure 1.2 is displayed as part of the musical texture (Kimmo Kuitunen: Trio for Clarinet, Cello and Piano, ). Figure 1.4 gives an example of ENP as an algorithmically generated user interface component. The two score views, shown inside a specialized PWGL application-box [42], are generated automatically based on information retrieved from an instrumental analysis database. Here, the most audible partials of the selected sound are displayed

22 22 1. Introduction using a detailed arpeggio view (left) and a more concise chord view (right). This box also contains other components that display amplitude envelopes and sound samples, for example. Publications [P6] and [P7] give further examples of how ENP can be used as an interactive user interface component. Figure 1.4: This example demonstrates the use of ENP as a GUI component. 1.4 The ENP Project The end result of this project is a portfolio that includes ENP for Macintosh OS X and Windows XP operating systems, this written report incorporating seven peer-reviewed articles, and an ENP Reference Manual and interactive tutorial both distributed as a part of ENP. Although it would have been possible to provide a stand-alone version of ENP for the purpose of this doctoral project, it is provided as a part of PWGL. There are several compelling reasons for this. First, ENP will never exist outside of the PWGL environment, and it would have been unreasonable to try to separate the two. Secondly, the

23 1.4 The ENP Project 23 unique potential of ENP is more clearly realized when it is used in tandem with the other software packages in PWGL. Furthermore, there is no ambiguity when it comes to distinguishing the author s work from that of the other developers of PWGL: everything relating to symbolic music notation is essentially the work of the author. PWGL, along with ENP, can be downloaded at The main part of this project involved software development. ENP itself is a combination of, among others, the following software components: 1. scripting language 2. visual editors (note head designer, expression designer) 3. rule based Find 4. plug-in architecture 5. note property editors 6. textual score description language 7. exporter (EPS, PS, MIDI, ETF, LilyPond, MusicXML). During this project, a considerable amount of effort was put into publishing activities. 28 papers (including two journal articles) have been published throughout this doctoral project. Most of these were written in collaboration with Mikael Laurson, although some articles were written jointly with researchers from the Acoustics Laboratory of Helsinki University of Technology. Mika Kuuskankare was the first author of 14 of the articles. These publications are listed below: International Scientific Journals 1. M. Kuuskankare and M. Laurson. Expressive Notation Package. Computer Music Journal, 30(4):67 79, Winter (in press). 2. M. Laurson, V. Norilo, and M. Kuuskankare. PWGLSynth: A Visual Synthesis Language for Virtual Instrument Design and Control. Computer Music Journal, 29(3):29 41, Fall 2005.

24 24 1. Introduction International Conference Proceedings 1. M. Castrén, M. Laurson, and M. Kuuskankare. MacSet: A Pcset-Theoretical Computer Application. In 26th Annual Meeting of the Society for Music Theory, Madison, K. Kuitunen, M. Kuuskankare, and M. Laurson. Large Scale Duration Organization in Hodoi tô Ergô. In Proceedings of International Computer Music Conference, pages , M. Kuuskankare and M. Laurson. ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, September M. Kuuskankare and M. Laurson. SpeCS, a Spectrogram-based Compositional System. In Symposium on Systems Research in the Arts IIAS, Baden-Baden, M. Kuuskankare and M. Laurson. ENP-Expressions, Score-BPF as a Case Study. In Proceedings of International Computer Music Conference, pages , Singapore, M. Kuuskankare and M. Laurson. ENP User-interface: Some Issues with Expert Systems in Music Notation. In Symposium on Systems Research in the Arts -IIAS, Baden-Baden, M. Kuuskankare and M. Laurson. Expressive Notation Package - an Overview. In International Symposium on Music Information Retrieval, M. Kuuskankare and M. Laurson. Intelligent Scripting in ENP using PWConstraints. In Proceedings of International Computer Music Conference, pages , M. Kuuskankare and M. Laurson. Recent Developments in ENP-score-notation. In Sound and Music Computing 04, October M. Kuuskankare and M. Laurson. Some Aspects on Music Scripting Languages. In Symposium on Systems Research in the Arts -IIAS, M. Kuuskankare and M. Laurson. Annotating Musical Scores in ENP. In International Symposium on Music Information Retrieval, 2005.

25 1.4 The ENP Project M. Kuuskankare and M. Laurson. ENP Expression Designer - a Visual Tool for Creating User Definable Expressions. In International Computer Music Conference, pages , M. Kuuskankare and M. Laurson. ENP Note Head Designer. In Journées d Informatique Musicale, M. Laurson and M. Kuuskankare. Instrument Concept in ENP and Sound Synthesis Control. In Journées d Informatique Musicale, Marseille, M. Laurson and M. Kuuskankare. PWGL: A Novel Visual Language based on Common Lisp, CLOS and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, September M. Laurson and M. Kuuskankare. Aspects On Time Modification In Score-Based Performance Control. In Proceedings of SMAC 03, pages , Stockholm, Sweden, August M. Laurson and M. Kuuskankare. From RTM-notation to ENP-score-notation. In Journées d Informatique Musicale, Montbéliard, France, M. Laurson and M. Kuuskankare. Some Box Design Issues in PWGL. In Proceedings of International Computer Music Conference, pages , Singapore, M. Laurson and M. Kuuskankare. Copy-synth-patch: A Tool for Visual Instrument Design. In Proceedings of International Computer Music Conference, pages 47 50, M. Laurson and M. Kuuskankare. PWGL Editors: 2D-Editor as a Case Study. In Sound and Music Computing 04, October M. Laurson and M. Kuuskankare. Extensible Constraint Syntax Through Score Accessors. In Journées d Informatique Musicale, M. Laurson and M. Kuuskankare. Micro textures with macro-notes. In Proceedings of International Computer Music Conference, pages , Finnish Conference Proceedings 1. H. Penttinen, J. Pakarinen, V. Välimäki, M. Laurson, M. Kuuskankare, and V. Norilo. Kiinalaisen kielisoittimen mallipohjainen äänisynteesi. In Akustiikkapäivät, Jyväskylä, Finland, November 2005.

26 26 1. Introduction Other Publications 1. M. Kuuskankare. Enpaattista nuottikirjoitusta. Tietoyhteys, (2), M. Kuuskankare and M. Laurson. Expressive Notation Package - a tool for music theoretical applications. Nordisk Musikforskarkongress, M. Laurson and M. Kuuskankare. PWGL: A Visual Programming Language for Computer Aided Composition, Music Analysis and Sound Synthesis. Nordisk Musikforskarkongress, ENP has also been an essential tool in two of the most recent research projects conducted at the Sibelius Academy: Sounding Score - Modeling of Musical Instruments, Virtual Musical Instruments and their Control and CLARA - Control Language for Audio Related Applications. ENP is among the main tools included in CLARA. It was used in these projects to produce musical demonstrations of several virtual instruments (e.g., guitar, clavichord, harpsichord, and guqin) developed by the team members. The author prepared the WWW pages ( describing both ENP and PWGL, and also lectured on ENP during the Prisma Course held at the Sibelius Academy, for example. He has given oral presentations and poster sessions on ENP and its applications at national and international conferences. These activities are listed below: Conference Presentations , International Computer Music Conference, Gothenburg, Sweden , Symposium on Systems Research in the Arts, Baden-Baden, Germany , Journées d Informatique Musicale, Montbéliard, France , 26th Annual Meeting of the Society for Music Theory, Madison, USA , International Computer Music Conference, Singapore , 5th International Conference on Music Information Retrieval, Barcelona, Spain.

27 1.5 The Scope of this Report , International Computer Music Conference, Miami, USA , Journées d Informatique Musicale, Paris, France , International Computer Music Conference, Barcelona, Spain , 6th International Conference on Music Information Retrieval, London, England. Other Presentations (Posters, etc.) , 14th Nordic Musicological Congress, Helsinki, Finland , 10th Symposium for Music Research in Finland, Turku. ENP has been tested primarily through the aforementioned activities. The publications have provided a scientific framework within which it has undergone the inspection of an international jury of peers, while both the technical and artistic demonstrations have provided another framework for testing the concepts and methods behind ENP in practice. 1.5 The Scope of this Report This report gives an overview of ENP and its applications. Although the potential of ENP as a music notation program is realized fully when used in tandem with PWGL, these applications are only partially covered here. Including all the PWGL applications would have yielded an unreasonably voluminous document, given the nature of this project, and it would have been less straightforward to identify the individual contributions of the members of the development team. As a result, however, several important applications (e.g., virtual instrument control) and a considerable amount of the author s work remain beyond the scope of the report. In addition to developing ENP and its applications during the four-year project,

28 28 1. Introduction the author has also been involved in developing PWGL, its different components, and the underlying concepts. This report only covers the notational front-end of ENP. The expression scheme is of primary importance in several of its applications, and most of the included articles deal with it in one form or another. The possibilities of annotating ENP scores, both manually and algorithmically, are discussed in detail. Furthermore, there is a special focus on a visual tool that could be used to create user-definable expressions. The remaining topics include the rule-based scripting language, ENP-Script, and the textual score description format, ENP-score-notation, both of which are important large-scale applications inside ENP. The former can be used to algorithmically perform various operations on an ENP score, and the latter for various purposes, including exchanging, describing and manipulating score-level information. Finally, visualization has been a major concern throughout the process of developing ENP. These concepts are present to a greater or lesser degree in all ENP applications. 1.6 The Contents of this Report This report is divided into two parts. The first part consists of an introductory essay and the second part of seven articles. The purpose of the introductory essay is to give supplementary examples in addition to those given in the articles. The articles, in turn, describe some of the most important concepts behind ENP. The first one [P1] gives an overview of ENP describing the underlying principles of internal data representation, the expression scheme, textual score notation, and scripting. The second article [P2], in turn, discusses the expression scheme in detail and provides several practical examples. Thirdly, [P3] gives an initial report of the scripting language, ENP-Script, and [P4] and [P5] together describe the creation and development of a text-based score representation, ENP-score-notation. Finally, [P6] and [P7] introduce two visual tools that can be used to design user-definable extensions to ENP.

29 1.7 The Structure of this Report 29 The latter two pairs of articles ([P4], [P5] and [P6], [P7]) in tandem give a comprehensive overview of their respective topics, ENP-score-notation and visual object design in ENP, which therefore are not covered in this introductory essay. 1.7 The Structure of this Report The publications (P1-P7) comprise the major part of this report. Chapter 2 of this Introductory Essay gives some complementary information on several aspects of notation. First, it illustrates through examples the notational abilities of ENP, secondly, it introduces a collection of typical ENP-expressions that show unique behavior when compared to the traditional approach taken in most notation programs, and thirdly, it provides several concrete applications of ENP-script. Chapter 3 offers some concluding remarks and suggestions for future directions, and finally, Chapter 4 gives a summary of the articles and the author s contribution. The rest of the report consists of the articles. 1.8 A Note on Terminology Some essential terms used throughout this report have somewhat different meanings than their counterparts in the common terminology of Western music. These terms are clarified here. Expression The term expression more specifically ENP-expression is used here to denote an ENP-specific device that can be used to represent multipurpose graphical objects as part of a musical texture. The traditional expression markings, such as articulations and dynamic markings, form a subset of ENP-Expressions. Chord and Note The components of an ENP score are enumerated and named in [P1] and presented here for convenience (see Figure 1.5). It is important to point out

30 30 1. Introduction that the terms chord and note are used in the context of object-oriented terminology: they are two different objects in the hierarchical structure of an ENP score. Chords are container objects holding an arbitrary number of notes. Notes, in turn, are complex objects holding, among other things, pitch and timing information. Moreover, both may contain information about expressions. SCORE PART VOICE(MENSURAL) MEASURE VOICE(NON MENSURAL) BEAT CHORD NOTE EXPRESSION Figure 1.5: An overview of the hierarchical object structure of an ENP score.

31 Chapter 2 Supplementary Examples Sections give supplementary examples in three important areas: notation and music representation (2.1), expressions (2.2) and scripting (2.3). 2.1 Notation The ENP approach to contemporary and most specifically graphical notation is radically different than the one found in other music notation software. The most significant difference is that the additional notational graphics in ENP are genuine objects. This has several advantages: (1) instead of being passive predefined figures, these graphical objects can adjust their visual appearance and position according to the musical context, e.g., pitch, time, loudness; (2) they may carry real musical significance, because they can contain any kind of data, e.g., annotations, other objects (subimages), information on timing or pitch; (3) the data they contain can be accessed by the user, e.g., the graphical information can potentially be converted into playback information Modern Notation By using ENP preferences it is possible to hide many different elements of notation either locally or globally. Figure 2.1 illustrates this technique. The score is written in 31

32 32 2. Supplementary Examples look-alike time notation using mensural (metric) notation. The score is written using invisible clefs, bar lines, stems, and beams. Each part is also rotated to create a notation that is similar to that used in some of the works produced in the 1960 s by Bussotti and others. It should be noted that, unlike in most other notation programs, the score given here is fully functional: it can be normally edited and played. Figure 2.1: An example of modern free notation in a mensural context.

33 2.1 Notation 33 Figure 2.2 shows an example of relatively complex graphical notation (a similar notation is to be found in Parentheses by J. Levine). In this case, a special ENPexpression, called Canvas-expression, is used to display the graphical information. Canvas-expression is a device that can be used to draw arbitrary graphical objects by constructing them out of a predefined selection of graphical primitives. Thus, it behaves like a miniature drawing program inside the score. Some additional examples of graphical notation are given in Figures 4 and 7 in [P1]. An interesting detail is that the graphical information shown here is generated automatically. A piece of Lisp code (some 60 lines of code) is used to construct the example containing a selection of randomly generated and placed graphical primitives, such as circles or filled polygons. All the other score elements except the Canvas-expression are hidden using ENP preferences.

34 34 2. Supplementary Examples Figure 2.2: A modern graphical score created using Canvas-expression (for a similar notation see, for example, Parentheses by J. Levine) Orchestral Notation Figure 2.3 gives an example of a score written for full orchestra. It demonstrates some of the automatic typesetting features of ENP: the grouping and ordering of the instruments, expression placement, and optimized systems (i.e., empty staves are hidden), for example. A contemporary look is achieved here by hiding empty measures and

35 2.1 Notation 35 shortening the staff lines in the case of partial measures. The Appendix in [P1] gives another orchestral excerpt taken from the literature. Figure 2.3: An orchestral score written in the modern style with partial and hidden measures (Mika Kuuskankare: Soave per Orchestra da Camera, 1996) Tape Notation ENP allows special notational styles to be used to change the visual appearance of the notation without modifying the underlying data structures. A special notational style can be applied to both mensural and non-mensural notation. Figure 2.4 shows an

36 36 2. Supplementary Examples alternative way of displaying normal mensural notation using a special style called tape notation. The part shown in parenthesis displays the underlying musical structure with its metrical rhythms and exact pitch. This kind of approach facilitates writing of music that contains discrete data (pitch and rhythm/time), but which is visualized using non-traditional or specialized notation. It also allows the musical data to be represented and manipulated normally: tape notation can be edited, annotated, and calculated, for example. The same principle could potentially be applied to shape-note, and tablature notation, for instance. Figure 2.4: An ENP part written using the tape notation style (top). The same part in normal notation (bottom) Music Analytical Notation ENP also allows for the representation of music analytical information, such as Schenkerian graphs. It currently follows the fairly traditional scheme according to which analytical notation is created by inputting the rhythmic and melodic structure and then embellishing the score with various markings (see Figure 2.5). However, ENP has in this case the following advantageous features: (1) the markings attached to the notation (beams, slurs, text), follow the notational objects they are associated with; (2) the additional markings are genuine objects, i.e., they may also contain additional information, such as textual notes; and (3) it is possible to algorithmically inspect, generate and modify the analysis information.

37 2.2 ENP-Expressions 37 Figure 2.5: A Schenkerian analysis graph realized with the help of ENP (adapted from [5]) It is also possible to split this kind of analysis information into layers, which allows one to manipulate and view one layer of information at a time and to see the potential hierarchical structure of the information more clearly. Figure 2.6 gives an example in which the layers are expanded into two different staff systems. Figure 2.6: The two layers found in the graph shown in Figure 2.5 displayed in their own separate staves. 2.2 ENP-Expressions This section gives examples of several ENP-Expressions that have unique features when compared to traditional expression markings (see [P1] for an introduction to the expression scheme). Figure 6 in [P1] introduces some typical ENP-expressions and illustrates some important concepts and behavior. [P2] also gives several practical applications, and a visual editor aimed at creating new expressions is in turn described in [P6]. Figure 2.7 gives an example that contains various standard expressions. Besides standard expressions, ENP is also capable of representing different kinds of specialized expressions. [P1] gives some examples of Score-BPF and Canvas-expression, both of

38 38 2. Supplementary Examples which are highly specialized expressions that can be used to visualize different kinds of graphical information as part of music notation. Figure 2.7: An ENP score containing various standard expressions, such as dynamics, accents, slurs, and tremolos (Mika Kuuskankare: Caractères II, for string quartet). One important feature of ENP-expressions is that they can be dynamic, i.e., they can react to the musical context in several different ways. In its simplest form this may mean avoiding collision with notes and chords, as is the case with most of the standard expression markings displayed in Figure 2.7. However, the side effect may also be more complex, e.g., changing the shape or size of the expression, or the information that is displayed. Moreover, expressions may also produce side effects in the notation itself. The following subsections enumerate ENP-expressions that have advanced behavior Lyrics Figure 2.8 gives an example of when the expressions produce side effects in the notational objects to which they are attached. Vocal music is traditionally beamed and flagged according to the syllables in the text. Through the use of the Lyrics-expression ENP can automatically beam vocal music according to the traditional rules, as shown in the second measure of Figure 2.8. Note, that this is melodically and rhythmically identical to the sixth measure, the only difference being that there is no lyrics information in the sixth measure and thus the latter is beamed as in instrumental music.

39 2.2 ENP-Expressions 39 Figure 2.8: Beams are broken automatically as appropriate when the user enters lyrics information into the score (Georges Bizet: Carmen) Dynamic Expressions Figure 2.9 shows a collection of ENP expressions that dynamically react to their musical context. They interpret and visualize the data contained in the notational objects with which they are associated. The PC-Clock, shown in Figure 2.9a, is an expression that dynamically visualizes the pitch-class set content of a chord using numeric notation (i.e., c=0, c =1). 1 The clock-face is arranged so that each hour represents a different pitch-class, 12 o clock being the first (0) and 11 o clock being the last (11). The highlighted (black) hours represent pitch-classes that are present in the chord and the grayed-out ones, in turn, represent those that are not present. 2 The guitar frame (Figure 2.9b), the trumpet fingering expression (Figure 2.9c) and the harp diagram (Figure 2.9d) are examples of expressions that are both dynamic and contain information about the associated instrument. The guitar frame is able to dynamically indicate on a fret diagram on which string and fret a note must be played. Trumpet fingering, in turn, shows the appropriate combination of three valves needed to play the corresponding note, and the harp diagram shows the required pedal positions. Figures 2.9e h show the same collection of expressions after the score has been transposed. The dynamic expressions have automatically updated the displayed in- 1 For information about pitch-class set theory and related terminology see [3]. 2 Due to pitch-class duplicates (e and e 2 ) the six pitches are reduced to five pitch-classes in the PCclock expression.

40 40 2. Supplementary Examples formation according to the pitch content of the objects to which they are attached. Figure 2.9: A collection of dynamic ENP-expressions: (a) a PC-clock, (b) a guitar frame, (c) a trumpet fingering, and (d) a harp pedal. The bottom line gives the score after it has been transposed. The aforementioned dynamic expressions can also be used to validate instrumental texture (see the highlighted items in Figure 2.10). The trumpet fingering expression indicates that the small octave f is not inside the regular range of the instrument (the C-trumpet in this case). 3 The last entry in the harp pedal expression, in turn, indicates that it is not possible to tune the B-string to both B and B simultaneously. Figure 2.10: The encircled items show that there are mistakes in the instrumental notation. 3 In this case, the normal range is used. The small octave f is the lowest natural pitch of the C- trumpet. However, the trumpet, like any brass instrument, is capable of playing so-called pedal tones, that extend its range below the normal.

41 2.2 ENP-Expressions Expressions Containing Break-point Functions There are some expressions that contain break-point functions (this information can be accessed for the purpose of playback, virtual instrument control, or algorithmic composition). These expressions include Score-BPF [P1], and all the agogics expressions (i.e., accelerando, ritardando, and rubato), and continuous dynamics expressions (i.e., crescendo and diminuendo). They can visualize themselves according to their current state. Normally, a crescendo is displayed as a traditional hairpin, and agogics as text, as shown in Figures 2.11a and b. Figures 2.11c and d, in turn, show the same expressions with their internal break-point functions visible. These expressions may also be displayed in an editable state (any object can be entered in an editable state by double-clicking). In this case the break-point function can be edited directly in place while the musical material remains in the background for reference. Figure 2.11: Two ENP expressions that internally contain a break-point function: (a) accelerando, and (b) crescendo. The lower score is the same, but in this case the expressions are drawn so that their internal break-point functions are shown (c and d).

42 42 2. Supplementary Examples Instrument-sensitive Expressions An ENP-expression may have a different graphical representation depending on the instrument it is written for. These kinds of expressions are called instrument-sensitive. Figure 2.12 gives an example of when an expression is used to denote that a passage of music is to be played on a certain string. For example, it is usually written differently in a guitar part than in a violin part, and if the user changes the instrument of a part, all of the instrument-sensitive expressions in it adjust their appearance accordingly. Figure 2.12: An ENP expression that is used to denote a string number for stringed instruments is able to adjust its visual appearance according to the instrument it is written for. The figure shows the same expression (string number 4) as written for Guitar (top) and Violin (bottom) Auto-canceled Expressions Auto-canceled expressions are special kinds of group expressions in ENP. They can be used to indicate various temporary changes in playing technique (e.g., pizzicato, col legno) or timbre (e.g., con sordino, sul tasto), for example. Auto-canceled expressions automatically mark both the beginning and the end of a passage with appropriate textual information. Traditionally, this kind of information is manually attached to the beginning of a given passage. Likewise, the appropriate cancelation marking is manually written where the passage ends. ENP, in contrast, requires only the affected passage to be marked, it then deals with the appropriate cancelation marking. There are several

43 2.3 ENP-Script 43 advantages in this: it is less prone to errors than the manual approach, and it is possible to copy part of a passage and have the playing technique copied with it. Figure 2.13 shows the basic behavior of a pizzicato expression. The ord. instruction (to change to ordinario or ordinary playing technique, i.e., bowing, in the case of the Cello) is the default cancelation marking for this particular expression. It is added automatically after the effect of the pizzicato expression has ended. Figure 2.13: Auto-canceled expressions provide a convenient way of indicating different playing techniques. ENP takes care of restoring the previous playing technique automatically. 2.3 ENP-Script ENP-Script is a scripting language that can be used to produce various side-effects on an ENP score. The scripts can also be applied only to a selection of notational objects (the selection can be made through the GUI). Its potential applications include: 1. adding expressions, 2. adding analytical information, 3. adjusting timing, 4. changing color, and 5. manipulating pitch and rhythm. ENP-Script uses a syntax that is identical to the PWGLConstraints patternmatching language. The ENP-Script section in [P1] covers the basic concepts and also gives a short presentation of the scripting syntax. The syntax is not explained here. Instead, the reader is advised to study [38] and [P3]. Furthermore, recent developments

44 44 2. Supplementary Examples in the syntax are covered in detail in [43], and the examples given below follow the new syntax presented therein. The following subsections provide examples of the applications enumerated above. The first three, given in Subsection 2.3.1, illustrate the use of ENP-Script as an input tool to automatize repetitive or routine tasks. Subsection shows how using a script enables visual aspects, such as size, position and appearance, of the notational objects to be manipulated, and Subsection considers ENP-Script as a music analysis tool Input Shortcut Inputting Pitch The aim in the next example is to enter the first two measures of Etude Op.25, No.10 by Frédérick Chopin. The passage contains chromatic texture played in octaves with several enharmonic alterations. The starting point, given in Figure 2.14, was created by manually inputting the first voice then duplicating and transposing it to create the second voice (by default, ENP writes both voices in the first staff of the staff system). Two scripts are given in Figures 2.16 and 2.17 respectively. The first one maps through all the chords in the passage (indicated by the :chord keyword in 2.16a). A new note, being an octave higher with the same enharmonic identity as the existing one, is inserted in each chord (2.16d h). The second script, in turn, maps through all the notes in the score including the ones inserted by the first script. If the pitch of a note is below a limit given in 2.17c it is assigned to the lower of the two staves (2.17d). 4 The resulting score can be seen in Figure In this case, two separate scripts are needed because the operations are not commutable. The properties of the notes cannot be checked before the new ones have all been inserted into the score, which is why it is imperative that the scripts are executed sequentially and in the correct order. 4 The staff numbering in ENP starts from 0.

45 2.3 ENP-Script 45 Figure 2.14: An initial input created by inputting the first voice then duplicating it and transposing it to create the second voice. The aim is to enter the first two measures of Etude Op.25, No.10 by Frédérick Chopin. Figure 2.15: The finalized input realized with the help of ENP-Script. The octaves and crossstaff changes are created automatically (Frédérick Chopin: Etude No.10, Op.25 in B Minor). a (*?1 :chord b (?if c (when (m?1 :complete-case? t) d (mapcar # (lambda(note) e (insert-note?1 f :midi (+ (midi note) 12) g :enharmonic (enharmonic note))) h (notes?1)))) i "Add octave(s) to a chord.") Figure 2.16: ENP-Script consisting of a scripting rule that inserts an octave in every chord in the score.

46 46 2. Supplementary Examples a (*?1 b (?if c (when (< (midi?1) 59) d (setf (clef-number?1) 1))) e "Assign qualifying notes to lower staff.") Figure 2.17: ENP-Script that assigns notes to the appropriate staves according to their pitch content Inputting Rhythm The script in Figure 2.20 can be used to change the time signatures of the score without changing the face value of the notes. It consists of two scripting rules. The first one (2.20a e) sets the denominator of each time signature to half of the original (2.20d), and the second one 2.20g k compensates the change of time signature by halving the unit-lengths of the main-beats (2.20i). The initial score is shown in Figure 2.18, and the modified result in Figure Figure 2.18: An initial score written with time signatures that have a quarter note as the denominator. Figure 2.19: The resulting score, after applying a script, that changes the denominators of the time signatures and the unit-lengths of the main-beats. This results in a score that has different time signatures but the same note face values as the original.

47 2.3 ENP-Script 47 a (*?1 :measure b (?if c (when (m?1 :complete-case? t) d (setf (low?1) (/ (low?1) 2)))) e "Halve the denominators of time signatures.") g (*?1 :beat h (?if i (when (m?1 :complete-case? t) j (setf (unit-length?1) (/ (unit-length?1) 2)))) k "Halve the unit-length of main-beats.") Figure 2.20: ENP-Script consisting of two scripting rules (a e and g k). The first rule changes the denominators of the time signatures, while the second one changes the unit-lengths of the main-beats. Inputting Expressions It is possible to use ENP-Script as a notational shortcut to insert additional attributes into the score. This is especially convenient when there is a repeating articulation pattern to be entered. Figure 2.21 shows a passage with a recurring articulation pattern consisting of two slurred notes followed by four notes with staccatos. Figure 2.22 gives the corresponding ENP-Script with two scripting rules. The given pattern is repeated on every downbeat (see 2.22b and g). The benefit of this use of a script is naturally more pronounced when the musical passage in question is very long. Figure 13 in [P1] gives an additional example of how to use ENP-Script as a shorthand to input expressions. Figure 2.21: The resulting score, after running a script that applies a constant articulation pattern. The slurs and staccatos are added automatically by the script shown in Figure 2.22 (Ludwig van Beethoven: String Quartet Op. 18, No. 6).

48 48 2. Supplementary Examples a (*?1?2 b (?if (when (downbeat??1) c (add-expression slur?1?2))) d "Add a slur to the notes.") f (*?1?2?3?4?5?6 g (?if (when (downbeat??1) h (add-expression staccato?3?4?5?6))) i "Add a staccato to the last four notes of the group.") Figure 2.22: ENP-Script showing two scripting rules (a d and f i) that define and insert an articulation pattern into an ENP score Score Manipulation The next example uses the beginning of Prélude Op. 28, No. 8 by Frédérick Chopin as a starting-point. In the original score the 32 nd notes in the right-hand staff are written using a smaller font. ENP accomplishes this by: (1) reducing the size of the beats in the upper voice, and (2) adjusting the stems so that they align correctly with the note heads in the lower voice. Making this kind of manipulation using the GUI would prove to be too laborious. Figure 2.25 gives the script used to perform the aforementioned editing steps. First, the size of the beats in the first voice (2.25a) is adjusted to 60% of the original (2.25d), then the second scripting rule 2.25g k is applied to shift the chords to the right (2.25j) to align the stems with the note heads in the second voice in the right-hand staff. Figure 2.23: A score displaying an excerpt taken from Prélude Op. 28, No. 8 by Frédérick Chopin. In the original score the fast figures (32 nd notes) in the right-hand staff are written with a smaller font.

49 2.3 ENP-Script 49 Figure 2.24: The score given in Figure 2.23 after running the script. Préludes Op. 28, No. 8). (Frédérick Chopin: a (*?1 :beat :part (1 1) b (?if c (when (m?1 :complete-case? t) d (setf (beat-scaling?1) 0.6))) e "Set beat size of a main-beat.") g (*?1 :chord :part (1 1) h (?if i (when (m?1 :complete-case? t) j (setf (x-offset?1) 0.2))) k "Move chords in the first voice by 0.2 units.") Figure 2.25: An ENP-Script that performs some score manipulation. The combined effect of the two scripting rules (a e and g k) is to set the size of the beats in the first voice to 60% of the original and to shift the chords slightly to the right to be aligned with the notes in the second voice. Next, an additional example in the context of non-mensural notation is given. Figure 2.26 shows an initial score containing chords that internally form written accelerandi and ritardandi gestures. In this example there are six different chord objects. The chords, in turn, are written so that the notes inside them have positive offset-times. In mensural notation this operation has no visual impact but in non-mensural notation the notes with non-zero offset-times are also visually displaced accordingly. As Figure 2.26 shows, the boundaries between different gestures cannot be easily perceived. However, in non-mensural notation, it is possible to display optional information (e.g., different kinds of beams) that makes it easier to identify the internal

50 50 2. Supplementary Examples structure of the chords. To make the excerpt given in Figure 2.26 more readable, an ENP-script is used to find and mark the structurally coherent gestures (see the script in Figure 2.28). Here, the internal offset-times of the notes are calculated and sorted in ascending order (2.28e h). Next, this information is interpreted and the appropriate structural information is inserted into the corresponding chord objects: lines 2.28i j determine if the ordered list of offset-times result in a decelerating gesture (i.e. the offset-times are increasing) and insert a ritardando beam (:rit-2) if appropriate; lines 2.28k l do the opposite, while 2.28m r inserts a normal beam (32 nd beam) in case all the offset-times inside the chord are within ±10 milliseconds of one another. Figure 2.27 gives the resulting score. Incidentally, the approximately constant offset-times found in the last case, could be used to determine the required number of beams, but three beams are used consistently in this example. Figure 2.26: The initial non-mensural score containing six chords (a f). The chords, in turn, contain notes with individual offset-times (relative to the start-time of the chord) forming accelrandi and ritardandi gestures.

51 2.3 ENP-Script 51 Figure 2.27: The resulting score with some structural information that has been inserted with the help of ENP-script. a (*?1 :chord b (?if c (when (m?1 :complete-case? t) d (when (rest (notes?1)) e (let ((dtimes f (pw::x->dx g (sort h (mapcar # offset-time (notes?1)) <)))) i (cond ((apply # > dtimes) j (setf (kind?1) :rit-2)) k ((apply # < dtimes) l (setf (kind?1) :acc-2)) m ((every # (lambda(x) n (<= (- (car dtimes) 0.01) o x p (+ (car dtimes) 0.01))) q dtimes) r (setf (kind?1) 3))))))) s "Determine and mark coherent gestures") Figure 2.28: ENP-Script is used to insert structural information into a non-mensural score according to the internal offset-times of the notes Analysis and Visualization Figure 2.29 gives an example of a classic voice-leading assignment that includes parallel fifths (copied from [20], p. 66), and figure 2.30 shows an ENP-script that detects and marks all parallel fifths in a musical texture.

52 52 2. Supplementary Examples Let us examine the script in more detail. First, 2.30c e ensure that both harmonies are complete, i.e., in the four-part texture there are four notes in each of the two harmonies. In 2.30f x the code loops through all the two-voice combinations. Lines 2.30n r verify that: (1) both consecutive intervals are perfect fifths; and (2) both voices move parallel to each other. Finally, 2.30s x inserts red line-shaped expressions between the notes in both offending voices. Figure 2.29 shows the automatically annotated score revealing the parallel fifths. For further examples of automatic music analysis and visualization see [P2] and Figure 15 in [P1]. Figure 2.29: A voice-leading assignment containing some (forbidden) parallel fifths (after De La Motte). The mistakes are marked directly in the musical texture with the help of a script.

53 2.3 ENP-Script 53 a (*?1?2 :harmony b (?if c (let ((c1 (m?1 :complete-case? t :object t)) d (c2 (m?2 :complete-case? t :object t))) e (when (and c1 c2) f (loop for p1 from 1 to (1- (length c1)) do g (loop for p2 from 2 to (length c2) h for c1/n1 = (find p1 c1 :key # voicenum) i for c1/n2 = (find p2 c1 :key # voicenum) j for c2/n1 = (find p1 c2 :key # voicenum) k for c2/n2 = (find p2 c2 :key # voicenum) l while (and c1/n1 c1/n2 c2/n1 c2/n2) m when n (and (= (abs (- (midi c1/n1) (midi c1/n2))) 7) o (= (abs (- (midi c2/n1) (midi c2/n2))) 7) p (/= (midi c1/n1) (midi c1/n2)) q (= (- (midi c1/n2) (midi c1/n1)) r (- (midi c2/n2) (midi c2/n1)))) do s (add-expression score-expression c1/n1 c2/n1 t :kind :line u :color :red) v (add-expression score-expression c1/n2 c2/n2 w :kind :line x :color :red)))))) y "Find and mark parallel fifths.") Figure 2.30: A relatively complex ENP-Script implementing a voice-leading checker that finds and marks parallel fifths in a given score.

54 54 2. Supplementary Examples

55 Chapter 3 Conclusions 3.1 Summary This report provides an overview of ENP, which was developed to meet the needs of computer-aided composition, music analysis and synthesis control. Its main purpose is to provide a front-end for a visual environment, PWGL. It has a mouse-driven graphical user interface and enables musical scores to be enriched with various notational attributes, ranging from standard articulation markings to user definable ones. It also allows for the generation of scores using its textual score description format. A score can be converted to the textual format and vice versa. This makes it possible to use PWGL to manipulate musical data in a flexible way. Moreover, ENP allows scripting through a specialized language, ENP-Script. This scripting scheme is useful when used simply as a speedy entry to attach articulations or articulation patterns, for example, but it can also be used to perform more sophisticated operations including music analysis and visualization. This ENP project was undertaken to develop methods and algorithms related to computer-aided music notation. So far it has yielded the following results: 1. the ENP music notation program; 2. a report summarizing the main components of ENP; 55

56 56 3. Conclusions 3. the ENP Reference Manual; 4. twenty-eight publications; 5. a web page ( describing ENP and its host application PWGL; 6. applications for computer-aided music notation in the following areas: composition, music research, music analysis, sound synthesis, and musical performance; 7. the development of new interaction and input strategies for musical notation dealing with the aforementioned applications; 8. better knowledge and expertise within the research area. 3.2 Future Directions ENP is a work in progress and it is inevitable that there are missing or incomplete features. Several improvements, new features and applications are planned Short-term Enhancements Typesetting Engine There is currently no visual synchronization between the two fundamental notational styles, mensural and non-mensural, which means that events in two parts written in mensural and non-mensural notation do not horizontally coincide. This is a visualization problem that should be addressed. The next project is to rewrite the typesetting engine to support synchronization between the notational styles, and to implement an intelligent page layout scheme.

57 3.2 Future Directions 57 Optimization There are also some other design details to consider. ENP currently recomputes the entire display after every modification made to the underlying musical structures. This makes the screen update demanding for the computer. Computationally more efficient incremental display models have been suggested, for example, by Dannenberg ([16, 17]). Although there is always room for optimization, computers are becoming so fast that radical optimization efforts may soon no longer be relevant. Missing or Incomplete Features The Undo feature is currently implemented only for destructive operations (e.g., deleting, changing rhythms, adding and removing expressions). This scheme should be finalized and stabilized to include transposing and, in general, all the potential UI operations. There are no supported import formats at the moment, and MIDI import, at least, is of primary importance. There exist GPL licensed MIDI packages, such as MidiShare ([47] and [23]), and Allegro [18], that implement MIDI file I/O. There is also another Free Software library, LibMusicXML [22], which implements both MusicXML export and import (incidentally, this would also allow ENP to export Guido [50] music format). High-Quality Fonts There is no native font support in OpenGL. There are basically three approaches to rendering fonts in OpenGL: bitmap, outline, and texture mapped. The texture map based technique was used to produce the fonts needed for ENP, but this kind of proprietary approach is both hard to maintain and difficult to interface with other systems, such as Postscript, that rely on real glyph information. Currently, there are free software packages, such as FreeType2 [1], for producing and managing high-quality font information. FreeType2 is a software font engine that provides an API to access font

58 58 3. Conclusions content, and supports a wide array of different font formats including TrueType and OpenType. FTGL [45], in turn, provides OpenGL bindings to FreeType. FTGL is a library for rendering TrueType fonts in OpenGL and it uses the FreeType2 API to extract glyph information from standard fonts. It would need some work to re-implement the font scheme in ENP in order to take advantage of both FreeType and FTGL. However, the benefits in moving to these solutions would be significant. As FreeType offers real kerning information it would be easier to produce EPS or PostScript printouts, for example. Both these packages are Free Software, and are also cross-platform compatible. Expanding the Expression Scheme Currently the ENP-expressions can be attached only to notes or chords. The present scheme should be extended to apply to all the objects in the score. This would make it possible to attach annotations to parts, voices, measures, bar lines, and clefs, for example, and it would also make the ENP object representation scheme more unified and coherent New Avenues of Research Computer-Aided Composition The musical problems that are interesting from the perspective of musical constraints programming are typically very demanding. Harmony, melody, voice-leading, and counterpoint provide individual and natural challenges in themselves. Defining a collection of rules to solve a certain compositional problem is a time-consuming task and requires a lot of musical and programming expertise. The PWGL tool for computerassisted composition, PWGLConstraints, provides excellent possibilities for addressing these kinds of problems. One area for potential improvement, however, is in how these rules are written by the user, and ENP could provide an alternative solution. It would be interesting to see how rules could be described by using the language most

59 3.2 Future Directions 59 familiar to musicians music notation. Computer-Assisted Music Analysis Computer-assisted music analysis is one of the most interesting ENP applications. It should be possible to provide intelligent expressions that are especially tailored to a certain analytical framework (e.g. Schenkerian analysis). Furthermore, for the purposes of computer-aided music analysis a stronger representation of analytical information is needed. It should be possible to project different views of a piece of music, i.e., to allow the notation to be displayed in any number of reductions or alternate forms. This kind of approach would be a powerful tool in computer-assisted music analysis. Computer Assisted Instruction Computer-Assisted Instruction (CAI) refers to a system of educational instruction performed almost entirely by computer. There are many advantages to using computers in educational instruction: they provide one-to-one interaction with the student and instantaneous response to the problem at hand, and they also allow students to proceed at their own pace. The facilities in ENP, including its integration with PWGLConstraints and PWGLSynth, make it an ideal CAI tool. Sophisticated tutors for basic music theory, solfège, post-tonal music theory, interactive harmony, ear training and other assignments could potentially be created with the help of ENP to replace the traditional Drill-and-Practice music theory programs. Content Authoring Tools Content Authoring Tools (CAT) provide teachers with the means to create new educational material (music theory exercises, solfège assignments, interactive tutors, ear training, etc.). They are used to arrange different kinds of visual and textual components to create interactive training content. They usually also support the linking of

60 60 3. Conclusions material into modules and provide visual and textual feedback for the user. It is also important for the modules and objects to be available for reuse in a modular fashion. ENP could provide such tools, including a language that allows the speedy construction of interactive pedagogical programs. The language should be simple enough to be learned by those without a programming background, but also powerful enough to represent advanced learning modules. Web applications There are numerous sites that offer Web applications targeted at basic-level music education (e.g., and These applications usually use Macromedia Flash or Java applets to provide some or all of the user interface components. There have been a few attempts to create plug-ins for rendering music notation in an Internet browser: FigureRenderer [36] can be used to display Lily- Pond as part of a web page; Niffty [2] is is a Java applet that allows NIFF [26] formatted music notation to be displayed on a Web page; and GUIDO NoteServer [29] displays GUIDO as common music notation on a Web page. There are also commercial Web plug-ins, such as musicrain 2.0 ( to facilitate the viewing of MusicXML. It would be an interesting field of study to see how ENP could be integrated with the World Wide Web. In its simplest form it could mean developing an ENP viewer plug-in that could be used to represent static notational information generated by ENP. A more advanced version would obviously involve some level of editability. Making ENP Web-ready would at least require charting the various Web-based formats (such as Scalable Vector Graphics, SVG) and technologies to identify the most viable solutions.

61 Chapter 4 A Summary of the Publications Publication [P1] [P1] provides an overview of ENP. The article describes an expression scheme that allows to enrich musical scores with various notational attributes ranging from standard articulation markings to user definable ones. [P1] also explains the principles of a textual score description format that allows the generation of scores. A score can be easily converted into the textual format and vice versa, which makes it possible to use PWGL to manipulate musical data in a flexible way. [P1] further describes a specialized scripting language, ENP-Script. This is a scripting scheme that is useful when used simply as a speedy entry for attaching articulations or articulation patterns, for example, but it can also be used to perform more sophisticated operations including music analysis and visualization. The author of this report wrote the article. Publication [P2] [P2] gives an overview of ENP-expressions, and presents a number of built-in expressions that are suitable for visualizing music analytical information as a part of music notation, for example. A system for creating user-definable annotation information 61

62 62 4. A Summary of the Publications is also described, and some sample algorithms are given. Finally, a system for automatically analyzing and annotating an ENP score is illustrated through examples. The author of this report wrote the text, implemented the expression devices, and prepared the examples. PWGLConstraints has been augmented with new keywords allowing richer access to the score by Mikael Laurson, in collaboration with the author of this report. Publication [P3] [P3] demonstrated how a constraint-based language, PWGLConstraints, can be adapted to be used to script ENP scores. The paper also presents and examines in detail a number of example scripts. The author of this report developed the idea of rule-based scripting, and wrote all of the paper, except Section 2. Publication [P4] The focus in [P4] is on how score information is represented in PWGL. A list-based notation, used in PatchWork (a predecessor of PWGL), is expanded so as to allow the generation of very detailed scores that contain not only the basic rhythmic structures but also additional score information. The author of this report wrote Sections 3 and 4. Publication [P5] [P5] describes ENP-score-notation, a text-based score representation format. The advantages in the text-based representation of a musical score are that the layout of the notational objects is separated from the representation, and that the textual formats are human-readable. In this case the textual format provides a way to generate and manipulate musical data in PWGL. The author of this report conceived ENP-score-

63 63 notation, which on the beat level was derived from the PatchWork RTM-notation but has been enriched in various different ways. The author of this report also wrote the paper. Publication [P6] [P6] introduces a tool called ENP Expression Designer (ED). It is aimed at visually creating new notational attributes to be used in ENP. Both visual and text-based interfaces are provided, and it is also possible to mix these approaches. Furthermore, the new expressions are viewed and designed in a real musical context. These concepts make the current system unique compared to other approaches. The author designed and implemented the ED editor and wrote the paper. Publication [P7] [P7] presents a visual editor called ENP Note Head Designer (ND), which was developed to assist both novice and expert users to design custom note heads in ENP. The same design principles apply as in the case of ED. The author of this report designed and implemented the ND editor and wrote the paper.

64 64 4. A Summary of the Publications

65 References [1] Freetype. [2] Niffty. [3] Allen Forte. The Structure of Atonal Music. Journal of Music Theory, [4] C. Ariza. An Open Design for Computer-Aided Algorithmic Music Composition. PhD thesis, New York University, New York, NY, USA, [5] D. Beach, editor. Aspects of Schenkerian Theory. Yale University Press, [6] A. Belkin. Macintosh notation software: Present and future. Computer Music Journal, 18(1):53 69, [7] P. Bellini, P. Nesi, and G. Zoia. Symbolic music representation in mpeg. IEEE MultiMedia, 12(4):42 49, [8] D. Blostein and L. Haken. Justification of printed music. Communications of the ACM, 34(3):88 99, [9] R. S. Brindle. The New Music : The Avant-garde since Oxford University Press, [10] W. Buxton, S. Patel, W. Reeves, and R. Baecker. Scope in interactive score editors. Computer Music Journal, 5(3):50 56, [11] D. Byrd. Music Notation by Computer. PhD thesis, Computer Science Department, Indiana University,

66 66 References [12] D. Byrd. User interfaces in music-notation systems. In Proceedings of International Computer Music Conference, pages , [13] D. Byrd and E. Isaacson. A music representation requirement specification for academia. Computer Music Journal, 27(4):43 57, [14] A. Cooper. About Face. The Essentials of User Interface Design. IDG Books, Foster City, CA, [15] R. Dannenberg. Music Representation Issues, Techniques, and Systems. Computer Music Journal, 17(3):20 30, [16] R. B. Dannenberg. A structure for representing, displaying and editing music. In Proceedings of International Computer Music Conference, pages , [17] R. B. Dannenberg. A Structure for Efficient Update, Incremental Redisplay and Undo in Graphical Editors. Software-Practice and Experience, 20(2): , [18] R. B. Dannenberg. Allegro Music Representation Language. edu/~rbd/software/allegro.htm, [19] R. B. Dannenberg, D. Rubine, and T. Neuendorffer. The resource-instance model of music representation. In Proceedings of the 1991 International Computer Music Conference, pages , [20] D. de la Motte. Harmonielehre. Bärenreiter-Verlag, [21] G. R. Diener. Modeling Music Notation: A Three Dimensional Approach. PhD thesis, Stanford University, [22] D. Fober, S. Letz, and Y. Orlarey. Open source tools for music representation and notation. In Sound and Music Computing 04, 2004.

67 REFERENCES 67 [23] D. Fober, Y. Orlarey, and S. Letz. Midishare joins the open source softwares. In ICMA, editor, Proceedings of the International Computer Music Conference, pages , [24] M. Good. Musicxml for notation and analysis. The Virtual Score: Representation, Retrieval, Restoration, pages , [25] J. S. Gourlay. Spacing a line of music. Technical Report OSU-CISRC-10/87-TR35, Ohio State University, Columbus, Ohio, [26] C. Grande. Beyond MIDI The Handbook of Musical Codes, chapter The Notation Interchange File Format: A Windows-compliant Approach, pages The MIT Press, Cambridge, Massachusetts, [27] H. Honing. Issues in the representation of time and structure in music. Contemporary Music Review, 9: , [28] H. Hoos, K. Hamel, K. Flade, and J. Kilian. Guido - music notation - towards an adequate representation of score level music. In Journées d Informatique Musicale, [29] H. H. Hoos, K. A. Hamel, and K. Renz. Using Advanced GUIDO as a Notation Interchange Format. In Proceedings International Computer Music Conference, pages International Computer Music Association (ICMA), San Francisco, CA, [30] D. Huron. Music information processing using the humdrum toolkit: Concepts, examples, and lessons. Computer Music Journal, 26(2):15 30, [31] J. T. M. III and S. M. Ornstein. Mockingbird: A composer s amanuensis. BYTE, 9(1): , [32] E. Isaacson. What you see is what you get: On visualizing music. In International Symposium on Music Information Retrieval, pages , London, UK, 2005.

68 68 References [33] E. Karkoschka. Das Schriftbild der Neuen Musik. Hermann Moeck Verlag, Celle, [34] K. Kuitunen, M. Kuuskankare, and M. Laurson. Large Scale Duration Organization in Hodoi tô Ergô. In Proceedings of International Computer Music Conference, pages , Barcelona, Spain, [35] M. Kuuskankare and M. Laurson. ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, [36] C. Lamb. FigureRender. [37] K. Lassfolk. Music Notation as Objects. PhD thesis, University of Helsinki, [38] M. Laurson. PATCHWORK: A Visual Programming Language and some Musical Applications. Studia musica no.6, doctoral dissertation, Sibelius Academy, Helsinki, [39] M. Laurson, C. Erkut, V. Välimäki, and M. Kuuskankare. Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis. Computer Music Journal, 25(3):38 49, Fall [40] M. Laurson and M. Kuuskankare. PWSynth: A Lisp-based Bridge between Computer Assisted Composition and Sound Synthesis. In Proceedings of the International Computer Music Conference, pages , Havana, Cuba, [41] M. Laurson and M. Kuuskankare. PWGL: A Novel Visual Language based on Common Lisp, CLOS and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, [42] M. Laurson and M. Kuuskankare. Some Box Design Issues in PWGL. In Proceedings of International Computer Music Conference, pages , Singapore, 2003.

69 REFERENCES 69 [43] M. Laurson and M. Kuuskankare. Extensible Constraint Syntax Through Score Accessors. In Journées d Informatique Musicale, Paris, France, [44] M. Laurson and M. Kuuskankare. Micro Textures with Macro-notes. In Proceedings of International Computer Music Conference, pages , Barcelona, Spain, [45] H. Maddock. FTGL. [46] H.-W. Nienhuys and J. Nieuwenhuizen. LilyPond, a system for automated music engraving. In XIV Colloquium on Musical Informatics (XIV CIM 2003), Firenze, Italy, [47] Y. Orlarey and H. Lequay. MidiShare: a Real Time multi-tasks software module for Midi applications. In ICMA, editor, Proceedings of the International Computer Music Conference, pages , [48] B. W. Pennycook. Computer-music interfaces: A survey. Computing Surveys, 17(2): , [49] G. Read. Music Notation. Victor Gollancz Ltd., [50] K. Renz. Algorithms and Data Structures for a Music Notation System based on GUIDO Music Notation. PhD thesis, Technischen Universität Darmstadt, [51] K. Renz. An Improved Algorithm for Spacing a Line of Music. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, [52] C. Roads. The Computer Music Tutorial. The MIT Press, Cambridge, Massachusetts, London, England, [53] Sibelius Software Ltd. Using the ManuScript Language. [54] L. Smith. Beyond MIDI The Handbook of Musical Codes, chapter SCORE, pages The MIT Press, Cambridge, Massachusetts, 1997.

70 70 References [55] G. L. Steele. Common LISP : The Language. Digital Press, 2nd edition, [56] K. Stone. Music Notation in the Twentieth Century: A Practical Guidebook. W. W. Norton & Company, [57] University of Leeds. Case Study: Representation of Musical Structure for Music Software, Leeds, September Music Notation Workshop: XML based Music Notation solutions. [58] T. Weyde and J. Wissmann. Visualization of musical structure. In Proceedings of the Conference on Interdisciplinary Musicology, Graz, [59] M. Woo, J. Neider, T. Davis, and D. Shreiner. OpenGL Programming Guide. Addison Wesley, Massachusetts, USA, 3rd edition, 1999.

71 Part II Publications 71

72

73 Changes in [P6] and [P7] The possibility of using textures and transparency when designing note heads as described in [P7] is removed from ENP, because of incompatibility with PostScript. PostScript does not support true alpha-blending. Furthermore, ENP cannot currently convert OpenGL textures to PostScript. Since the introduction of the two editors described in [P6] and [P7] there have been some changes in terms of appearance and functionality. Currently, these tools are highly interactive and provide instant visual feedback. Any change made is immediately reflected in the appearance of the designed object. When the user changes the code in the code-view, the effect is shown in the preview score/canvas. The user is not required to perform manual updates, as was the case before. The tools also provide feedback as to the syntactic correctness of the code. 73

74 74

75 Errata In [P4], page 5, first paragraph,... without scarifying the functionality should be sacrificing In [P7], page 6, section Conclusions, first paragraph,... any of the build-in ones. should be built-in 75

76

77 Publication [P1] [1] M. Kuuskankare and M. Laurson. Expressive Notation Package. Computer Music Journal, 30(4):67 79, Winter (in press). 77

78 Mika Kuuskankare * and Mikael Laurson *Department of Doctoral Studies in Musical Performance and Research Centre for Music and Technology Sibelius Academy Helsinki, Finland {mkuuskan,laurson}@siba.fi Expressive Notation Package At present, there are a large number of commercial and free programs dealing with music notation. Two of the most notable free software projects are Lily- Pond (Nienhuys and Nieuwenhuizen 2003) and Guido (Hoos et al. 1998; Renz 2002). Both of these programs are LaTeX-like languages that are used to describe the contents of a musical score in a textual form and not through a point-and-click user interface. Furthermore, a Web site dedicated to music notation software, ace.acadiau.ca/score/others.htm, lists a large number of other commercial and free music notation programs including simple notation editors (e.g., abc) for typesetting relatively simple notation; special-purpose editors like Grégoire for typesetting Gregorian chant, Django for typesetting tabulature, and GOODFEEL for Braille notation; and full-featured programs like Berlioz, Igor (programmed in Lisp), Nightingale, and SCORE. Finally, the two de facto commercial notation programs are Finale and Sibelius. Recently, Web-viewable applications have also started to emerge. There are a few commercial approaches such as Scorch by Sibelius Software. ScoreSVG (Bays 2005), in turn, is a free alternative based on the Scalable Vector Graphics, a language for describing two-dimensional vector graphics in XML. A handful of Lisp-based applications also exist that are aimed at representing musical data, such as Common Music Notation (CMN; Schottstaedt 1997), the Rhythm-Editor of PatchWork (RTM; Laurson 1996), and the musical editors in OpenMusic (Assayag et al. 1999). One of the earliest experiments, MUZACS (Kornfeld 1980), was even written for a Lisp machine. Of these editors, RTM is primarily intended to represent musical raw material, and thus the editing capabilities are limited. This is also the case with Open Music s notation editors. CMN is powerful typesetting package but lacks a graphical user interface (GUI). Computer Music Journal, 30:4, pp , Winter Massachusetts Institute of Technology. Expressive Notation Package (ENP; Kuuskankare and Laurson 2002) is a music notation program that has been developed to meet the requirements of computer-aided composition, music analysis, and synthesis control. ENP is programmed with Lisp- Works Common Lisp ( Lisp- Works, in turn, is a Lisp implementation that is source-code compatible across Windows, Linux, Mac OS X, and some UNIX platforms. ENP is primarily intended to represent Western musical notation roughly from the 17th century onward with a strong emphasis on modern notation. ENP is not a full-featured music typesetting program. However, it is designed to produce automatic, reasonable musical typesetting according to the common practices described, for example, in Stone (1980) and Read (1982). ENP is used as a notational front end in a visual programming language called PWGL (Laurson and Kuuskankare 2002). PWGL is a combination of several software packages build on top of Common Lisp. In addition to ENP, the components include, for example, a rule-based programming language called PWGLConstraints (Laurson 1996), and a sound-synthesis program called PWGLSynth (Laurson et al. 2005). The development of PWGL is focused toward the Mac OS X operating system, but it currently runs also under Windows XP. A Linux version is under consideration. PWGL is freeware, and it can be downloaded from our Web site at ENP has a sophisticated GUI based on direct editing, a rich set of musical primitives, an extended concept of expressions, and algorithmic control over scores. It contains a unique set of features that cannot be found elsewhere, at least to this extent of integration in one program. ENP allows one to analyze, modify, view, and annotate scores in many different ways; a text-based format can be used to generate scores algorithmically; it is possible to modify data contained by the objects in a score by using a scripting language; our rule-based programming language can be used Kuuskankare and Laurson 67

79 to calculate the pitch content of an ENP score; and finally, ENP can be used to represent and generate control information for our software synthesizer. Owing to its dynamic nature, ENP is primarily intended for composers and music analysts. The system has also proven useful in various research activities dealing with physics-based musical instrument design and control. Furthermore, potential applications that are of interest in the near future are various educationally related activities that require intelligent feedback from the computer, such as computer-aided teaching of counterpoint, harmony, and solfege. The present article summarizes some of the most important concepts behind ENP. We concentrate on four main topics: the underlying data representation, the expression scheme, the ENP-scorenotation format, and the ENP-Script language. These topics have also been discussed in separate publications. The data representation has been covered briefly in Kuuskankare and Laurson (2002). The expression scheme has been discussed in general in Kuuskankare and Laurson (2003, 2005). ENPscore-notation has been reported in both Laurson and Kuuskankare (2003) and Kuuskankare and Laurson (2004b). Finally, ENP-Script is introduced in Kuuskankare and Laurson (2004a). This article intends to elaborate on these topics and to provide larger and more comprehensive examples. Music Representation in ENP Understanding the structure of an ENP score is of primary importance, because it is reflected in every aspect of ENP: its GUI, expression scheme, scripting, etc. There is an excellent introduction to many different representation schemes in Selfridge-Field (1997). Music representation issues in general have also been discussed, for example, in Diener (1990), Huron (1992), Honing (1993), Dannenberg (1993), Roads (1996), and Lassfolk (2004). An ENP score is built out of hierarchical object structures. There are two fundamental notational styles: mensural and non-mensural notation. Typically, a mensurally notated ENP score contains a number of parts; each part contains voices; each voice contains measures; and each measure can contain any number of beats. Beats, in turn, can contain other beats (to create complex rhythmic structures) or chords. Finally, chords can contain an arbitrary number of notes. Notes and chords can also contain information about additional attributes such as expressions. Figure 1 shows a short musical excerpt written in a mensural style along with the corresponding hierarchical tree structure of the score. In the tree view, objects are represented by their ENP class name (e.g., NOTE means an ENP note object). The numbers following the class names have two different meanings. In case of beats, they indicate the proportional unit length of the beat, and in the case of notes, they indicate the MIDI note number. The last beat in the score and the corresponding part in the object structure are both marked with a gray rectangle. The beat-level ENP score representation is based on the concept of PW-beat (Laurson 1996). PW-beat itself is a hierarchical structure that can be used to represent complex metrical rhythms. Each beat has two important properties: the unit-length and potential sub-beats. By manipulating these properties, it is possible to represent a comprehensive set of rhythms sufficient for most practical purposes. Figure 2 gives detailed examples of some rhythms and their corresponding structural representations. Again, the unit lengths of the beats are written inside parentheses after the class name of the object. The main unit in this case is always a quarter note. (See the first beat on the top left.) Although the beat representation is closely related to the one used in PatchWork, the score representation, in general, is considerably different from that of PatchWork. ENP object structure now reflects the traditional structure of a (paper) score, with parts, voices, instruments, key signatures, expressions, etc. Moreover, the rhythmic structures have been augmented to include grace notes, for example. The difference in non-mensural notation compared to mensural notation is that internally, a nonmensural voice contains a collection of chords with absolute start-times and durations. The nonmensural notation can be used, for example, when writing contemporary time-notation (proportional notation). Figure 3 gives an example of an ENP score that is written using non-mensural notation. 68 Computer Music Journal

80 Figure 1. An ENP score (bottom) and the corresponding tree view of the ENP score structure (top). (Igor Stravinsky, Rite of Spring, reproduced by kind permission of Boosey and Hawkes Publishers, Ltd.) Figure 2. Two different representations of three rhythmic structures in ENP. Figure 1 Figure 2 The fundamental notational styles in ENP behave differently from the approach typically used in other notation software, whereby the user writes time notation in the normal mensural context. In the latter case, only the appearance of a time notation is achieved by removing certain graphical components of the notation, such as stems, bar lines, etc. The internal structure of the data does not reflect the fact that the notation is free of any metrical structures. In ENP, by contrast, the structure of Kuuskankare and Laurson 69

81 Figure 3. A score written in contemporary time notation. There is no underlying metric structure present; instead, musical events are notated with absolute onset times and durations, proportionally spaced. Figure 4. An example of modern time notation in a mensural context. A lineshaped expression is used to connect the invisible (hidden) note heads to mimic the string glissando writing of, for example, Krzysztof Penderecki. the data is reflected directly in the notational output. It also determines how the user interface behaves: There are no measure boundaries, and the notational objects can be freely dragged in time. However, by adjusting ENP preferences, it is possible to hide many different elements of notation either locally or globally. Figure 4 gives an example using this technique: a score written in time notation using mensural (metric) notation. The lineshaped expressions found in this example are attached to notes that they connect. Thus, the expressions logically follow the pitch contour if the score is, for example, transposed. This kind of behavior is also different from the existing approaches where graphical information is often considered passive in nature. Next, we briefly demonstrate the concept of voices in ENP. As discussed previously, a part can contain an arbitrary number of voices. A part can be drawn using one of the available systems, which, in turn, can contain single or multiple staves, each having their own clefs, etc. A voice can then be drawn either on one individual staff, or it can temporarily cross staves within the system. Figure 5 shows an organ part containing four voices that are written in a typical organ system consisting of three staves. Two voices share the topmost staff, and the remaining two occupy the other staves. The Appendix gives an example written for full orchestra: the first page of Elliot Carter s composition A Mirror on Which to Dwell. Expression Scheme The notational attributes, ENP-Expressions, are of primary importance in terms of our software. Here, we use the term expression in a broader sense than a traditional expression marking. ENP-Expressions are multipurpose visualization devices that can be used to represent complex dynamic Lisp-based objects as part of a musical texture. In addition to their traditional use, ENP-Expressions can be used in a wide range of applications: display of musictheoretical analysis information, such as annotation of motives and harmonic progressions; visualization of specialized analytical information, such as Schenkerian graphs and pitch-class-set theoretical information; attachment of arbitrary textual annotations, names, or comments to objects; and dynamic inspection and visualization of data contained by the notational objects, such as pitch, interval duration, velocity, and onset time. They can also be used to modify the notational output itself. (See Figure 4 for an example.) ENP provides a predefined set of both standard and non-standard expressions. Standard ENP- Expressions include, for example, staccatos, slurs, accents, and playing styles. Non-standard expressions include, among others, Group, Canvasexpression, and Score-BPF (Kuuskankare and Laurson 2003). Of these, Canvas-expression is a multipurpose graphical device that can be used to draw graphical objects in a score. Score-BPF, in turn, is a specialized group expression that can represent break-point functions as a part of a musical texture. ENP also offers a visual editor that can be used to create new expressions through a GUI. (See the section ENP Expression Designer. ) ENP-expressions can be dynamic, i.e., they can adjust their visual appearance according to the nota- 70 Computer Music Journal

82 Figure 5. A part containing four voices that are distributed in three different staves (Max Reger, Phantasie und Fuge über B-A-C-H, Op. 46, copyright Universal Edition A.G., Wien / UE 1222, Figure 6. (a) Score containing a collection of ENP- Expressions, e.g., (1) Pitch-Class Clock, (2) Marker, (3) Score-BPF (breakpoint function), and (4) Group; (b) the same material transposed a minor sixth higher. The ENP- Expressions automatically adjust their position and the displayed information accordingly. (a) (b) tional context. Expressions are also editable directly in the score; therefore, adjusting the slope of a slur or repositioning an articulation can be done interactively. Figure 6a gives a simple score containing various ENP-Expressions. Figure 6b, in turn, shows the same score after it has been transposed up by a minor sixth. Canvas-Expression is a device that can be used to draw graphical objects by constructing them out of a predefined selection of graphical primitives. Currently, these primitives include text objects, lines, arrows, filled or framed polygons, and circles. Canvas-Expressions behave as miniature drawing programs inside the score. Figure 7a shows an example of a Canvas-Expression that is in an editable state, i.e., all the shapes inside the dark canvas can be moved, reshaped, added, and deleted. Figure 7b, in turn, shows how relatively complex graphical notation can be entered with the help of Canvas- Expression. This particular example mimics the writing that can be found, for example, in nonstop for piano (1960) by Boguslaw Shaeffer. Score-BPF is a specialized form of a group expression. It is a graphical object that can represent any number of break-point functions as a part of a musical texture. Score-BPFs can be used to describe pitch contours, tempo functions, envelopes, dynamic curves, etc. Potentially, the same mechanism could be used to represent samples, sonograms, and even spatial paths. Score-BPF can be edited directly in the score. It also allows the contained break-point functions to be modified and generated algorithmically. Figure 8 shows an application of Score-BPF. Here, control information specialized to a virtual Guqin instrument model has been drawn in the score with the help of several Score-BPFs. The example also contains special tabulature markings (below the staff) that are used to indicate the string numbers one to seven. (The Guqin is a traditional Chinese instrument that has seven strings.) These characters are based on a new computer-based coding system developed by Henbing Li (2005). This coding system, in turn, is based on an ancient Chinese tabulature notation called Jian Zi Pu (JZP-notation; Changlin 1989). ENP Expression Designer Modern music in particular often requires special expression markings. Many historical playing techniques exist, and new ones are invented all the time. For example, Karkoschka (1966) and Risatti (1973) alone list thousands of different modern notational devices. Against this background, it becomes evident that it is an impossible task for a music notation program to make available a comprehensive collection of expression markings. It is more reasonable to provide an interface that lets the user build custom expressions. Two of the most widely used commercial notation programs, Finale and Sibelius, provide the possibility to create user-defined expressions. In CMN, it is also possible to define new expressions using Lisp (here, user-definable graphical entities are called sundries). Furthermore, LilyPond includes a \markup facility a typesetting mini language and it also allows users to define their own expressions in C++. However, when using these tools, it is not possible to view and design the expressions in the context of music notation, i.e., to see during the design process how the new expression is going to appear in the score. ENP Expression Designer (ED) is a visual tool that can be used to create custom expressions in ENP. ED provides a protocol that helps users build new Kuuskankare and Laurson 71

83 Figure 7. (a) Canvas- Expression shown here is in an editable state, showing the collection of available tools; (b) a graphical score, in the style of Boguslaw Shaeffer, created using Canvas-Expression. (a) (b) view. ) The expression is constructed out of six Bézier curves that form an opening fan that fills the horizontal space reserved for the note. ENP-Score-Notation expressions. This approach enables the following: (1) expressions can be made dynamic, because they are defined using Lisp code; (2) expressions are displayed in a fully functional score that allows the user to observe how the expressions behave, e.g., when the music is transposed (an approach different from any existing one); and (3) it is possible to combine both algorithmic (code-based) and visual (graphical) approaches. ED is part of an increasing collection of visual tools inside ENP. Currently, these tools also include Color Editor, Expression Tool Editor, and Note Head Designer. The purpose of these tools is to provide a visual interface for constructing complex music-notation-related Lisp objects and to provide assistance to those users who do not possess the needed programming skills to use the strictly textbased approach. The visual approach, however, is not for inexperienced users only. Some objects may be realized more easily by using a purely graphical approach and others, in turn, by using a programming language. Uniform design principles have been applied to all the aforementioned tools. The tools share the same kinds of editors, components, and functionality, which makes them easier to learn. Most editors also contain an interactive view providing visually synchronized feedback. The use of visual editors in our case has several advantages: (1) the underlying Lisp syntax can be hidden to allow the user to concentrate on the task at hand; (2) the system automatically handles such tasks as class and method definitions; and (3) this approach is less prone to errors, because most of the underlying code is generated automatically. Figure 9 shows an example of a typical ED session. Here, we have used the code-based approach to define an expression. (See the view labeled Code Many different text-based formats have been developed for representing the information in a musical score. Some typical examples are CMN, GUIDO, MusiXTEX (Taupin, Mitchell, and Egler 2005), GNU LilyPond, and MusicXML (Good and Actor 2003). Some Lisp-based representations are also able to preserve only a subset of musical information, such as the rhythm; these include the notation formats of both PatchWork and OpenMusic. The advantages in text-based representation of a musical score are that the layout of notational objects is separated from the representation, and that the textual formats are human-readable. In our case, the textual format also provides a way to generate and manipulate rich musical data in PWGL. ENP allows users to describe scores in a special text-based format called ENP-score-notation (Laurson and Kuuskankare 2003). It is offered as an intermediate step between the score and the low-level file format. By using ENP-score-notation, a score can be converted into a more readable form that can also easily be converted back to an ENP score. This kind of symmetry is beneficial, as it allows users to view and modify score information without losing information. The structure of ENP-score-notation reflects the score-hierarchical structure of the score, and the syntax is similar to Lisp list syntax. Every level in the hierarchy is collected into a list. ENP-scorenotation offers a large subset of notational attributes. Through keyword arguments, the system also allows users to add new features easily. Next, we give an example of ENP-score-notation and the resulting ENP score (see Figure 10). We have chosen here a slightly unorthodox visualization scheme to be able to align the corresponding elements of the two representations. The score is displayed on the right, rotated 90 clockwise, and the ENP-score-notation code is shown in the middle. On the left is a rhythmic reduction, resulting in 72 Computer Music Journal

84 Figure 8. Score-BPF expression provides the possibility to draw piecewise linear functions as a part of a musical texture. In this example several Score- BPFs are used to describe pitch glides for a Chinese string instrument called Guqin. Note also the Chinese tabulature markings below the staff. Figure 9. ED is used to define an expression (see Code view) that represents a fanning glissando in which six different instruments start from unison and make gestures that approximately follow the shape of the expression. Figure 8 Figure 9 Kuuskankare and Laurson 73

85 Figure 10. ENP-scorenotation example (middle); the corresponding score (right); the rhythmic reduction (left). The structural elements in the ENP-score-notation are aligned with and connected to the matching sections in the musical score. (The alto saxophone is notated here at sounding pitch.) Figure 11. A fragment of ENP-score-notation is broken into three different segments. The aggregate effect of adding optional attributes is shown in the music notation below. (a) Rhythmic skeleton; (b) pitch information; and (c) optional attributes, such as expressions and enharmonic spelling. the unit-lengths of the main-beats. This is also aligned with the corresponding elements in the textual notation. Figure 11 attempts to illustrate two important concepts concerning ENP-score-notation: defaults and optional attributes. Here, an ENP-scorenotation fragment taken from Figure 10 (lines 3 10) is divided into three columns. The basic idea behind this visualization is that, by starting from the rhythmic skeleton shown in the leftmost column, the score can be described in more detail by adding the information found in the two succeeding columns. The result of the rhythmic skeleton alone (specifying only rhythmic elements, including potential rests and ties) is shown in the bottom of the first column. As can be seen, the score already contains information about pitch, tempo, and other attributes that are not indicated in the textual description. ENP automatically substitutes missing properties with default values (e.g., the pitch is 74 Computer Music Journal

86 middle C, the clef is a treble clef, and the tempo is M. M. = 60). Furthermore, some properties (such as time signatures) are always calculated automatically by ENP. The second column (b) adds pitch information to the rhythmic skeleton. The more detailed result can be seen again in the bottom of the column. For example, there is now a note with an alteration (sharp) as ENP spells all altered notes as sharps unless specified otherwise. Finally, in the last column (c), the information in the score is completed with some additional attributes and ENP-Expressions. The score fragment in the last column shows the aggregate of all three columns. Here, the :enharmonic attribute is used to indicate that the associated note is written using its lowered enharmonic equivalent. The result can be seen in the corresponding score fragment: The A- sharp found in the previous column has been changed to B-flat. ENP-Script Most professional music notation programs provide a scripting language or plug-in interface that allows the user to modify notational information contained in the score. In a music notation program, a script can typically be used to apply a certain articulation pattern to a passage of music or to recalculate the enharmonic identity of selected notes. Sibelius, for example, provides a built-in scripting language called ManuScript. Finale, on the other hand, provides an interface that allows the user to write plugins using C++. It is also possible to use the scripting language called FinaleScript. LilyPond provides a plug-in functionality through its built-in Scheme interpreter. ENP-Script (Kuuskankare and Laurson 2004a) is a scripting language that uses a pattern-matching syntax identical to the one used by PWGLConstraints. The use of the pattern-matching language as the basis of our scripting language offers several advantages: (1) complex musical patterns can easily be defined; (2) the syntax is compact, powerful, and easy to learn; (3) typically there is no need to write any control structures (e.g., loops) owing to the basic ability of the system to map through the notational objects in the score; and (4) recently introduced enhancements to the pattern-matching syntax (Laurson and Kuuskankare 2005) allow ENP-Script to map through different types of objects in the score in addition to notes (e.g., measures, beats, chords, and harmonies). Furthermore, ENP-Script is intended to be open, allowing full access to the notational structures of the score. Scripts can access and manipulate a rich set of melodic, rhythmic, and harmonic information found in the score. Next, we give two examples of how ENP-Script can be used in a musical context. The complete syntax of the language is not presented here; instead, the basic concepts are covered to make the included scripting examples more comprehensible. The syntax of the pattern-matching language is covered in detail in Laurson (1996) and Laurson and Kuuskankare (2005). In the first example, we use an excerpt taken from Arnold Schönberg s Suite, Op. 25, as a starting point. We begin with a score that contains only pitch and rhythm information. The aim is to fill this excerpt with some of the articulations that are found in the original score with the help of ENP- Script. To make the example more straightforward, we exclude all articulations other than grave accents and slurs. We attempt to determine the type and position of the inserted articulations by accessing the notational information contained in the musical objects. This is done according to the following two features: (1) the size of the melodic movement; and (2) the beaming level. The original score features a slur between two notes if they form an interval larger than an octave and if there is not a rest between them. Also, each sixteenth note has a grave accent. Without going into details, we give the definition of an ENP-Script consisting of two scripting rules. Figure 12a shows the first scripting rule that traverses all adjacent melodic note pairs in the score and determines if a slur should be inserted according to the user-defined conditions. Figure 12b, in turn, gives the second scripting rule that handles grave accents. In both cases, the lines starting with add-expression insert the appropriate expression Kuuskankare and Laurson 75

87 Figure 12. An ENP-Script, consisting of two scripting rules, is used to insert articulations into a score according to (a) pitch and (b) beaming information contained by the notational objects. (See the comments on the right following semicolons.) Figure 13. Slurs and grave accents found in this example are inserted by an ENP-Script. The type and placement of the expressions has been determined according to the beaming information and the size of the melodic interval. The example contains only a subset of the articulations found in the original score (Arnold Schönberg, Suite, Op. 25, copyright 1925 by Universal Edition A.G., Wien/UE 7627, ENP is a work in progress, and several features are still missing or incomplete. There is currently no visual synchronization between the mensural and non-mensural notation. This means that, for example, events in two parts written in mensural and non-mensural styles do not coincide even if they have exactly the same start-time. Also, the pagelayout capabilities of ENP are currently quite primitive. The next project is to rewrite the notation engine to support synchronization between the notational styles and to implement an intelligent page-layout scheme. ENP allows users to export several graphics and music file formats, including EPS, PS, MIDI, ETF, LilyPond, and MusicXML. LilyPond and MusicXML formats are currently implemented through FO- MUS (Psenicka in press) by David Psenicka. However, there are no supported import formats at the moment; here, MIDI is of primary importance. Curinto the score. Figure 13 shows the score after the script has been applied. Next, we show how to automatically analyze and annotate some harmonic and melodic properties of a choral piece, in this case Erotessa by the Finnish composer Kimmo Kuitunen, who writes: The cycles of three four-voiced symmetric chords and their inversions (the consecutive set-class identities are 4 7, 4 7 and 4 9 respectively) create the harmonic basis. The pitch classes remain the same within each four repetitions of the cycle, but the octave positions change. These three chords (F A E A-flat, D B-flat E-flat B, and D-flat G-flat G C) also form a matrix that fills the chroma (set class 12-1). The melodic line of each vocal part repeats the three-note cells derived from the matrix. (Set classes are either 3-3a or 3-3b). Each voice also uses all twelve pitch classes, thus filling the chroma. (Kuitunen 2006) This kind of analysis requires a high level of expertise. It is also very time-consuming and prone to errors. Here, instead of doing the analysis by hand, we use ENP-Script and a set of specialized graphical devices to visualize the analysis information directly in the score. (See the result in Figure 15.) We concentrate on two aspects of the analysis given above. Figure 14a shows the scripting-rule to locate and visualize the symmetric harmonies, and Figure 14b analyzes the set-class identity of consecutive three-note cells. The first of these scripting-rules is seemingly more complex than the ones in the previous case, owing to the visual appearance of the analysis information that is calculated inside the scripting-rule. (The appearance is cycled through three different shapes: circle, rectangle, and triangle.) These shapes are used to distinguish between consecutive symmetric harmonies that potentially share notes. Thus, for example, the third harmony (marked with triangles) shares two notes (found in the middle voices) with the next harmony that, in turn, is marked with rectangles. Future Work 76 Computer Music Journal

88 Figure 14. An ENP-Script with two scripting rules is used to insert (a) vertical and (b) horizontal analysis information into a score. Figure 15. All symmetric harmonies are displayed automatically in the score with the help of connected shapes. In addition, the set-class content (3-3a or 3-3b) of consecutive threenote cells is shown below each part (Kimmo Kuitunen, Erotessa, with lyrics by Eino Leino, for mixed choir, 2001, reproduced by kind permission of the composer). rently, we are considering Allegro (Dannenberg 2000), which is a text-based representation that, in conjunction with a converter tool (allegroconvert) could potentially be used to both export and import MIDI files. There exists also a library, LibMusicXML (Fober, Letz, and Orlarey 2004), that is licensed under the GNU Lesser General Public License (LGPL) and suitable for both exporting and importing MusicXML format. (Incidentally, it would also support Guido.) Acknowledgment The work of Mikael Laurson has been supported by the Academy of Finland (SA ). References Assayag, G., et al Computer-Assisted Composition at IRCAM: From PatchWork to OpenMusic. Computer Music Journal 23(3): Bays, G ScoreSVG: A New Software Framework for Capturing the Semantic Meaning and Graphical Representation of Musical Scores Using Java2D, XML, and SVG. Master s Thesis, College of Arts and Sciences, Georgia State University. Changlin, C Preliminary Application of the Computer Technique to GuQin Music Research. Journal of Computer Science and Technology 4(1): Dannenberg, R Music Representation Issues. Computer Music Journal 17(3): Dannenberg, R Allegro Music Representation Language. Available online at software/allegro.htm/ (accessed May 25, 2006). Diener, G. R Modeling Music Notation: A Three- Dimensional Approach. Ph.D. Thesis, Stanford University. Fober, D., S. Letz, and Y. Orlarey Open-Source Tools for Music Representation and Notation. Paper presented at Sound and Music Computing 04, Paris, 21 October. Good, M., and G. Actor Using MusicXML for File Interchange. Third International Conference on WEB Kuuskankare and Laurson 77

89 Delivering of Music. New York: Institute of Electrical land Electronics Engineers, p Honing, H Issues in the Representation of Time and Structure in Music. Contemporary Music Review (9): Hoos, H. H., et al The GUIDO Music Notation Format A Novel Approach for Adequately Representing Score-level Music. Proceedings of the 1998 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Huron, D Design Principles in Computer-Based Music Representations. In A. Marsden and A. Pople, eds. Computer Representations and Models in Music. London: Academic Press. Karkoschka, E Das Schriftbild der Neuen Musik. Celle: Hermann Moeck Verlag. Kornfeld, W Machine Tongues VII: LISP. Computer Music Journal 4(2):6 12. Kuitunen, K correspondence with the author, 14 June Kuuskankare, M., and M. Laurson ENP2.0: A Music Notation Program Implemented in Common Lisp and OpenGL. Proceedings of the 2002 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Kuuskankare, M., and M. Laurson ENP- Expressions, Score-BPF as a Case Study. Proceedings of the 2003 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Kuuskankare, M., and M. Laurson. 2004a. Intelligent Scripting in ENP using PWConstraints. Proceedings of the 2004 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Kuuskankare, M., and M. Laurson. 2004b. Recent Developments in ENP-score-notation. Paper presented at Sound and Music Computing 04, Paris, 21 October. Kuuskankare, M., and M. Laurson ENP Expression Designer: A Visual Tool for Creating User- Definable Expressions. Proceedings of the 2005 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Lassfolk, K Music Notation as Objects. Ph.D. Thesis, University of Helsinki. Laurson, M PATCHWORK: A Visual Programming Language and Some Musical Applications. Ph.D. Thesis, Sibelius Academy, Helsinki. Laurson, M., and M. Kuuskankare PWGL: A Novel Visual Language based on Common Lisp, CLOS and OpenGL. Proceedings of the 2002 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp Laurson, M., and M. Kuuskankare From RTMnotation to ENP-score-notation. Paper presented at Journées d Informatique Musicale 2003, Montbéliard, France, 4 June. Laurson, M., and M. Kuuskankare Extensible Constraint Syntax Through Score Accessors. Paper presented at Journées d Informatique Musicale 2005, Paris, 2 June. Laurson M., V. Norilo, and M. Kuuskankare PWGLSynth: A Visual Synthesis Language for Virtual Instrument Design and Control. Computer Music Journal 29(3): Li, H The Encoding of Guqin Jian Zi Pu Tablature for Computer Input. Unpublished manuscript, Institute for Psychoacoustics and Electronic Music, Department of Musicology, Ghent University, Ghent, Belgium. Nienhuys, H.-W., and J. Nieuwenhuizen Lily- Pond: A System for Automated Music Engraving. XIV Colloquium on Musical Informatics. Firenze, Italy: Centro Tempo Reale. Psenicka, D. (in press). FOMUS, a Music Notation Package for Computer Music Composers. Proceedings of the International Computer Music Conference. Read, G Music Notation: A Manual of Modern Practice. New York: Taplinger. Renz, K Algorithms and Data Structures for a Music Notation System based on GUIDO Music Notation. Ph.D. Thesis, Technischen Universität Darmstadt. Risatti, H. A., New Music Vocabulary: A Guide to Notational Signs for Contemporary Music. Champaign- Urbana: University of Illinois Press. Roads, C The Computer Music Tutorial. Cambridge, Massachusetts: MIT Press. Schottstaedt, B Common Music Notation. In E. Selfridge-Field, ed. Beyond MIDI: The Handbook of Musical Codes. Cambridge, Massachusetts: MIT Press. Selfridge-Field, E., ed Beyond MIDI: The Handbook of Musical Codes. Cambridge, Massachusetts: MIT Press. Stone, K Music Notation in the Twentieth Century: A Practical Guidebook. New York: Norton. Taupin, D., R. Mitchell, and A. Egler MusiXTex. Available online at directory.fsf.org/musixtex.html/ (accessed October 19, 2005). 78 Computer Music Journal

90 Figure 16. Typeset with the ENP software, this score excerpt shows measures 1 2 of Elliott Carter s A Mirror on Which to Dwell for soprano and chamber orchestra, a setting of poems by Elizabeth Bishop. Copyright 1976 (renewed) by Associated Music Publishers, Inc. (BMI). International copyright secured. All rights reserved. Reprinted by permission. Appendix A page of an orchestral score created with ENP (Elliot Carter, A Mirror on Which to Dwell, 1975) is shown here as Figure 16. Kuuskankare and Laurson 79

91 Publication [P2] [P2] M. Kuuskankare and M. Laurson. Annotating Musical Scores in ENP. In International Symposium on Music Information Retrieval, London, UK,

92 ANNOTATING MUSICAL SCORES IN ENP Mika Kuuskankare Department of Doctoral Studies in Musical Performance and Research Sibelius Academy Finland Mikael Laurson Centre for Music and Technology Sibelius Academy Finland ABSTRACT The focus of this paper is on ENP-expressions that can be used for annotating ENP scores with user definable information. ENP is a music notation program written in Lisp and CLOS with a special focus on compositional and music analytical applications. We present number of built-in expressions suitable for visualizing, for example, music analytical information as a part of music notation. A Lisp and CLOS based system for creating user-definable annotation information is also presented along with some sample algorithms. Finally, our system for automatically analyzing and annotating an ENP score is illustrated through several examples including some dealing with music information retrieval. Keywords: notation. Music representation, annotating, symbolic 1 OVERVIEW Expressive Notation Package (ENP, [1, 2]) is a music notation program that is designed for displaying scores using the common Western music notation. ENP has been used in several research projects ranging from computer aided composition to controlling virtual instruments. A special focus has been given to compositional and computerassisted music analysis applications. ENP has a graphical user interface that allows musical objects to be edited directly with the mouse. It supports two fundamental notational styles, i.e., mensural and non-mensural notation, and a number of special notational styles such as time notation, frame notation, etc. Representation of musical units must offer ways of making annotations, giving names, making comments, adding images or diagrams, providing links to informative resources on the web, etc [3]. ENP provides a collection of standard and non-standard notational attributes Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. c 2005 Queen Mary, University of London (e.g, articulations) called ENP-expressions. Furthermore, it offers a set of attributes that can be used to represent analytical information or other user-defined annotations as a part of a musical texture. In addition to their traditional use, ENP-expressions can be used in wide range of applications: (1) display music theoretical analysis information, e.g, annotate motives, harmonic progressions, etc; (2) visualize specialized analytical information, such as Schenker graphs, or pitch-class set theoretical information; (3) attach arbitrary textual annotations, names, or comments to objects; (4) dynamically inspect and visualize data contained by notational objects, i.e., duration, velocity, start-time, etc.; (5) add instructions to tutorials, documentation, or presentations, etc. All ENP-Expressions can access the data contained by the notational objects they are associated with. This allows to design dynamic expressions that can automatically display relevant information about themselves and their musical context. It is also possible to use a scripting language called ENP Script [4] as an algorithmic complement to the manual approach where the user inserts ENPexpressions by hand. This is useful when building, for example, computer-assisted music analysis applications or automatically annotating a musical score. In the following, we present and discuss the annotation possibilities of ENP in more detail. We start with a brief introduction of ENP-Expressions and the notational output of ENP. Section 3 is the main part of this study. Here we present a wide range of annotation devices along with some example scores. Section 4 gives a brief look at the possibilities of automatic music information retrieval and annotation. Section 5 presents some conclusions and ideas for future work. 2 ENP-EXPRESSIONS IN BRIEF Every expression is attached to some musical object or to a group of objects. Currently these objects can be either notes or chords (see [5] for a description of the object hierarchy in ENP). All the expressions are aware of their musical context and can automatically adjust their position and graphical appearance accordingly. New expressions can be created through a textual interface using Lisp and CLOS or by using a set of specialized editors inside ENP. Figure 1 gives a concise overview of the notational possibilities of ENP in a modern context. The example is

93 written in non-mensural notation (time notation) and contains various expressions, special note heads and playing styles. Figure 3: Some static text expressions inserted in a score. The association between the expression and the notational object is displayed by a dotted line Figure 1: An example of the notational output of ENP. Another example of a static expression is the window expression. This device is useful for giving focus to an abject at any position on the screen. These expressions apply only to the current window so they are suitable for inserting markings only to one static page at a time, as in case of tutorials or presentations. 3.1 Groups 3 ANNOTATIONS OVERVIEW Groups are multi-purpose expressions that can be used to mark continuous passages in music. The visual appearance of the groups can be adjusted to suit the current application. Groups can also contain user definable textual information. The next example (Figure 2) shows one use for groups to annotate motives in a score. The groups, in this case, are represented as brackets with labels. Figure 2: Motivic analysis information inserted in the score with the help of groups (J.S. Bach: Invention no. 1). 3.2 Text Expressions Textual information can be attached to both chords and notes. The expressions can be either static or dynamic. These expressions typically contain some written instructions or annotations made by the user. Dynamic ones, in turn, can adjust the visual appearance and printed information by analyzing their musical context. Figure 4: Window expressions can be used to mark any position in the score Dynamic Text Expressions Dynamic expressions can adjust their visual representation (e.g, position, shape, color, text) automatically. This ensures that the score is always up-to-date. This is also useful when dealing with analytical problems: the analysis information remains correct even if the music is edited. For this purpose we introduce a special expression called annotation. Dynamic expressions are also useful to display relevant information accessed from the objects they are attached to. Almost any attribute of the notational objects can be made visible in this way. The attribute can be selected from an automatically generated list that is specialized for both chords and notes. The current set includes, for example, midi, start-time, diatone, duration, velocity, channel, etc. Figure 5 gives one example of the behavior of dynamic annotation. The data displayed is read directly from the objects. In this case the data shown is the value found in the midi slot of the note Static Text Expressions Next, we give two examples of static text expressions. In Figure 3 the user has inserted text expressions to the notes and entered the note names accordingly. This kind of behavior is acceptable when the data contained by the notes is not expected to change (e.g., through transposing). In this case the expressions show their association with the notes by drawing a dotted line connecting the expression to the corresponding note. Figure 5: Dynamic expressions can adjust their visual appearance automatically. The user can also write additional methods for displaying the result of some user-definable code with the help of annotation expressions. ENP provides an add-toannotation-library macro that can be used to add user definable annotations to a special library. These annotations are immediately usable and can be inserted into the score

94 through context sensitive menus. The parameters of addto-annotation-library are: (1) the type of the object the expression is associated with, (2) the name of the expression, and (3) the function that constructs the displayed data. Below we give two relatively simple examples of dynamic text annotations created with the help of the add-toannotation-library macro. First, we define a new annotation that can be attached to notes (1). In (2) we define the name of the annotation (note-name) and in (3) we give the function that constructs the textual information depending on the midi value of the note. Figure 6 gives the corresponding score. (add-to-annotation-library :note ;(1) note-name ;(2) # (lambda(note) ;(3) (format () "note name: a" (midi-to-note-name (midi note))))) Figure 6: A dynamic text annotation, displaying the note name, created with the help of the add-to-annotationlibrary macro. As can be seen in Figure 7, the needed user-interface components (context sensitive menus) are also added automatically. The annotations can be inserted in the score as any built-in ENP-expression (see the User methods menu in Figure 7). The two menus marked as Methods and Slots contain a list of available slots and predefined methods applicable to the object in question. Figure 8: The twelve-tone row of Alban Berg s Lyrische Suite (above) and its transposed version (below). Note, that the pitch classes shown by the dynamic annotations (pc-name) are displayed correctly in the transposed row form. 3.3 Score Expressions Score-expressions can be used to visualize discontinuous information in the score (Figure 9). This is a convenient way to display information that is scattered across different parts. Arbitrary vertical relations can be made visible for analytical purposes. The shape of the score expression can also be selected. The next example illustrates the use of the box-shaped score-expression to draw a focus around two chords in the score. This kind of an expression is useful for revealing groups containing several entries. Figure 9: Focus drawn around two chords using a box-shaped score-expression (Arnold Schoenberg: Sechs Kleine Klavierstücke, op. 19). Figure 7: Menus for user definable annotations are added automatically by the system. The second example deals with displaying some pitchclass set theoretical analysis information. Here we use a dynamic text expression to mark the pitch-classes above the individual notes in a twelve-tone row. In Figure 8 we have two twelve-tone rows. The former one is the row used in Alban Berg s Lyrische Suite in original form, and the latter one, in turn, is its transposed version. The expression definition is given below. There are only few changes when compared to the previous example. The most important one is that in this case we take the modulo 12 of the midi value contained by the note, as can be seen in (1). (add-to-annotation-library :note pc-name # (lambda(note) (format () " a" (mod (midi note) 12)))) ;(1) A line-shaped score-expression is best suited to indicate a relation between two notational objects as can be seen in the Figure 10. Here some music analytic information (cross relation) is inserted in the score. In addition to the line connecting the two notational objects the expression can contain some user definable text. Figure 10: Voice-leading analysis information displayed using the line-shaped score-expression. 3.4 Pitch-class Clock Face The Pitch-class clock is a highly specialized expression that can be used to visualize set theoretical information as a part of the musical texture. It is a dynamic graphical object that displays the pitch-class set of a selected chord as a pitch-class clock face. The pitch classes that are present in the set are highlighted (Figure 11).

95 Figure 11: A pitch-class clock face is a specialized expression for visualizing pitch-class set theoretical information. 3.5 Highlighting Highlighting is a visualization device that can be used for demonstration purposes. Relevant information in the score can be highlighted to focus the attention to a specific detail in the score (Figure 12). The areas outside the highlighted area are dimmed but still clearly visible, thus preserving the view to the whole musical context at the same time. The highlighted area can be moved and re-sized by the user. in the score (see the :harmony keyword in (1) in the example script). A special Lisp function add-expression (2) is used to insert into the score a group expression displaying the analysis information. The expression is attached to the lowest note of the chord (3). In (4) the built-in settheoretical function sc-name is used to identify the pitchclass set name of the current harmony. As can be seen in Figure 13 there are three pitch-class set names below the first note in the second measure (E ). This is due to the fact that there are three different harmonies that include the note in question. The set-class names are, from the bottom up: (1) 1-1, which is the result of the E sounding alone, (2) 2-5, which is the result of the E and two-line B in the right hand staff sounding together, and (3) 2-6, resulting from the low E and the one-line A in the right hand. (*?1 :harmony ;(1) (?if (when (complete-chord??csv) (add-expression ;(2) group (give-bass-item?csv) ;(3) :info (sc-name (m?1)))))) ;(4) Figure 12: A highlighted area created in the bassoon part of Rite of Spring by Igor Stravinsky. 4 AUTOMATICALLY ANNOTATING A SCORE In this section we briefly describe how to automatically analyze and annotate an ENP score by using a built-in scripting language, ENP script. A script is usually used to automate complex or advanced tasks within the program. In a music notation program it is useful to allow frequently repeated operations (e.g., applying an articulation pattern) to be automated by using scripts. The scripting examples in this section are explained only briefly. It is not our intention to explain the syntax of ENP Script but to illustrate the potential of the system in general. In our first example we examine the vertical aspect of a piece of music by performing a simple harmonic analysis. 1 The result of the analysis, the name of the pitch-class set of each prevailing harmony, is printed on the bottom of the score. The naming of the pitch-class sets follows the conventions introduced by Allen Forte [6]. Figure 13 gives a score with the set theoretical analysis information. The script used to generate the information is shown above the score. This kind of use of a script resembles the container-iterator idea found, for example, in CPNView [7, 8]. Here we iterate through every harmonic situation 1 Here we consider the term harmony to be a result of one, two or more notes being sounded simultaneously. Usually the smallest possible harmony contains at least two notes. For simplicity, monophonic situations are regarded as special cases. Figure 13: Vertical pitch-class set theoretical analysis information displayed in the score with the help of ENP Script and groups (Arnold Schoenberg: Suite für Klavier, op. 25). The second example deals with horizontal aspects of music. The script in this case is used to calculate and display the interval chain of a melodic line. The script is given below and the resulting score can bee seen in Figure 14. In this case we are interested in the relation between two consecutive notes (see (1) in the script). An analysistext-group expression is used in this case (2). It is positioned above the associated objects (see Figure 14). The expression is also displaced horizontally (3) so that it is positioned between the notes. The printed information is calculated by subtracting two consecutive pitches (4). (*?1?2 ;(1) (?if (add-expression analysis-text-group?1 ;(2) :user-position-x-correction ;(3) (- (/ (width?1) 2.0) 1.0) :info (- (m?2) (m?1))))) ;(4) Figure 14: The interval chain of the melodic line displayed above the staff (Anton Webern: Zwei Lieder, op. 8). Finally, we give an example dealing with n-grams. N- grams have been widely used in text retrieval, where the

96 sequence of symbols is divided into overlapping constantlength sub-sequences [9]. The use of n-grams in MIR has also been considered by several authors (see for example [10]). Let us examine the script in more detail. It is divided into two parts. The first part (A) calculates di-grams and the second part (B) calculates tri-grams. The Lisp function, add-expression, is again used to insert the analysis information into the notes (2). This time the information is attached to multiple notes (see the di-gram information and the corresponding brackets in Figure 15). Also a more subtle bracket style is utilized (3). In (4) and (5) the interval information is calculated and formatted (the specifics of the Lisp s format function are not in the scope of this paper. See [11] for detailed information). The second part of the script (B) is otherwise identical, when compared to the first one, except that we now consider four consecutive notes (6) forming three intervals. The printed information is again calculated in (7-9). A. (*?1?2?3 ;(1) (?if (add-expression group?1?2?3 ;(2) :kind :bracket-at-end ;(3) :info (format () " 3,@d 3,@d" ;(4) (- (m?2) (m?1)) ;(5) (- (m?3) (m?2)))))) B. (*?1?2?3?4 ;(6) (?if (add-expression group?1?2?3?4 :kind :bracket-at-end :info (format () " 3,@d 3,@d 3,@d" (- (m?2) (m?1)) ;(7) (- (m?3) (m?2)) ;(8) (- (m?4) (m?3)))))) ;(9) Figure 15: Di- and tri-grams displayed as a part of a musical texture (J.S. Bach: Musikalisches Opfer). 5 CONCLUSIONS A comprehensive set of different annotation devices in ENP were discussed. Also a scheme to automatically analyze a score and to visualize analytical data was introduced. Currently the ENP-expressions can be attached only to notes or chords. The present scheme should be extended to apply all the objects in the score. This way it would be possible to attach annotations to parts, voices, measures, bar lines, clefs, etc. It is possible, however, to insert comments to measures, for example, but these annotations have different status than the regular ENP-expressions. By addressing this problem it would also make the ENP object representation scheme more unified and coherent. Displaying MIR results in a score is suitable only for certain types of data. To bring ENP closer to practical MIR applications it would be beneficial to integrate a statistical package, such as R [12], into the environment (R is a language and environment for statistical computing and graphics). This would allow to represent, for example, pitch histograms and other similar information retrieved from an ENP score. ACKNOWLEDGEMENTS The work of Mikael Laurson has been supported by the Academy of Finland (SA ). REFERENCES [1] Mika Kuuskankare and Mikael Laurson. ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, September [2] Mika Kuuskankare and Mikael Laurson. Expressive Notation Package - an Overview. In International Symposium on Music Information Retrieval, [3] Tillman Weyde. Case Study: Representation of Musical Structure for Music Software. [4] Mika Kuuskankare and Mikael Laurson. Intelligent Scripting in ENP using PWConstraints. In Proceedings of International Computer Music Conference, pages , [5] Mika Kuuskankare and Mikael Laurson. Recent Developments in ENP-score-notation. In Sound and Music Computing 04, October [6] Allen Forte. The Structure of Atonal Music. Journal of Music Theory, [7] Donncha Ó Maidín. Common Practice Notation View: a Score Representation for the Construction of Algorithms. Proceedings of International Computer Music Conference, pages , [8] Donncha Ó Maidín and Margaret Cahill. Score Processing for MIR. International Symposium on Music Information Retrieval, pages 59 64, [9] Shyamala Doraisamy. Polyphonic Music Retrieval: The N-gram Approach. PhD thesis, University of London, [10] J. S. Downie. Evaluating a Simple Approach to Music Information Retrieval: Conceiving Melodic N- grams as Text. PhD thesis, University of Western Ontario, [11] Guy L. Steele. Common LISP : The Language. Digital Press, 2nd edition, [12] Francisco Cribari-Neto and Spyros G. Zarkos. R: Yet another econometric programming environment. Journal of Applied Econometrics, 14: , 1999.

97 Publication [P3] [P3] M. Kuuskankare and M. Laurson. Intelligent Scripting in ENP using PWConstraints. In Proceedings of International Computer Music Conference, pages , Miami, USA,

98 Intelligent Scripting in ENP using PWConstraints Mika Kuuskankare DocMus, Sibelius Academy Mikael Laurson CMT, Sibelius Academy Abstract In this paper we are going to focus on how a constraintbased language, PWConstraints, can be adapted so that is can be used to script our music notation package, called Expressive Notation Package (ENP). We briefly introduce the basic concepts behind PWConstraints and give an overview of the scripting properties of ENP. We also present and examine in detail a number of example scripts. 1 Background Scripting languages are programming languages specific to a particular program. They are usually weakly typed or untyped and interpreted rather than compiled. A scripting program, a script, is normally used to automate complex or advanced tasks within the program. Scripts typically interact either with other programs or with a set of functions provided by the interpreter. Scripts are usually stored in ASCII format, thus they can be distributed across different computer platforms. This is an important feature for cross-platform computer applications. A practical application of a script is to program either repetitive or otherwise laborious tasks so that they can be executed in a single pass. In a music notation program such a task could be to apply a certain articulation pattern to a passage of music, to recalculate the enharmonic identity of selected notes, etc. There are some commercial music notation programs that provide a scripting language and/or plug-in interface that allows the user to modify the information in the score. In Sibelius, for example, the user can write scripts by using the built-in scripting language called ManuScript. The ManuScript syntax resembles, in many aspects, Java and C++. ENP (Kuuskankare and Laurson 2002) is a music notation program that we have developed in order to meet the requirements of computer aided composition, music analysis and virtual instrument control. ENP is programmed with Lisp and OpenGL (Woo et al. 1999). It s purpose is to display scores using the traditional western notation. Furthermore, ENP provides a rich set of both standard and user definable notational attributes, called ENP-expressions, which range from standard articulation marks to a fully editable multi-purpose graphical expression, called score-bpf. All ENP-expressions are dynamic, i.e, they adjust their visual appearance according to their notational context. PWConstraints (Laurson 1996), in turn, is a family of constraint-based programming tools that are aimed at solving a wide range of constraint satisfaction problems. Constraint satisfaction is a general problem in which the goal is to find values for a set of variables that will satisfy a given set of constraints (Leler 1988). The musical applications of PWConstraints are closely related to ENP and PWConstraints can access the data structures behind the ENP notational frontend. In the following we compare some important features found in other scripting languages, such as ManuScript, to our scripting system in ENP. (1) Usually, when writing a script, the user has to write code to handle looping through the notational objects in the score. In our approach the user never sees the main loop as it is handled by PWConstraints. (2) In case of ManuScript, for example, the scripts are not undoable. In order to revert the effect of the script one must manually remove the attributes one by one. In ENP, on the other hand, the system records knowledge about the individual runs so that the user can later identify them and remove the attributes in any combination (selective undo). (3) In most of the music notation programs the notational attributes are not dynamic. Let us assume a case where the user adds some analysis information to the score with the help of a script. Whenever the related musical information changes the analysis results will become invalid. Normally, the user is required to remove the old attributes by hand and run the script again. In ENP, however, the attributes are dynamic (when applicable). Once inserted they adjust themselves according to the changes made in the music notation. (4) Most scripting languages lack the ability of PWConstraints to easily define and identify complex musical patterns with the help of a patternmatching syntax. The rest of the paper is structured as follows. First we

99 give an overview of PWConstraints and the pattern-matching language. Next we discuss the scripting in ENP and give an outline of the scripting properties. We end the paper with some example scripts and concluding remarks. 2 Overview of the PWConstraints PWConstraints is a general-purpose rule-based programming language where the user writes rules to describe the end result from a number of different viewpoints. A searchspace is defined as a set of search-variables. Each searchvariable has a domain containing a list of values. In a rule a pattern-matching language is used to extract relevant information from a potential solution. This information is given to a Lisp test function that either accepts or rejects the current choice made by the search-engine (i.e., the rules return a truth value). A PWConstraints rule consists of two parts: the patternmatching part and the Lisp-code part. A special symbol?if is used in the rule to distinguish between the pattern-matching part and the Lisp-code part. The pattern-matching part consists of several different pattern-matching variables. The variables are named as follows: 1. Named variable (e.g.,?1,?note1,?note2) is used to extract single search-variables from a partial solution. The name (the portion of the text after the question mark) is user definable. 2. Anonymous variable (?) is used when a search-variable is required to define a pattern but its value can be ignored. 3. Wild card (*) matches any continuous part of a partial solution. In the pattern-matching part the wild card can occur only once. 4. Index variable (e.g., i1, i12) extracts search-variables from an absolute position in the partial solution. The following gives an example of a PWConstraints rule. This rule forces an ascending series of numbers: (*?1?2 (?if (<?1?2))) The pattern-matching part contains a wild card and two named variables. The rule loops through every adjacent pair of search-variables (indicated by?1 and?2)) in the given search-space. The Lisp-code part, in turn, applies a test function (<?1?2) to each pair and returns a truth value. PWConstraints contains also a collection of more specialized tools that extend the behavior of the basic scheme described above (Laurson and Kuuskankare 2001). These tools include Score-PMC, Texture-PMC, and MA-PMC (MA stands for music analysis). Score-PMC is used to solve polyphonic search problems. An input score (an ENP score with the rhythmic structure prepared in advance) is given as an argument to the search-engine. The search, in turn, aims at filling the input score with pitch information according to the given rules. Texture-PMC, in turn, is used to produce rhythms and textures. Finally, MA-PMC is used to analyze existing rhythmic, melodic and harmonic material in a score. Unlike the former cases MA-PMC does not use backtracking. In all aforementioned tools a search-variable represents a note in the input score. In these cases the search-variables contain knowledge about pitch, rhythm, harmony, etc. This knowledge can be accessed in a rule with a set of specialized Lisp-functions. In this study, MA-PMC is used as a starting point to implement our scripting language. This approach is further explained in the following Section. 3 Scripting in ENP The scripting language of ENP uses MA-PMC to provide the basic ability to map through the notational objects in the score. ENP script can contain an arbitrary number of scripting rules. In case of the scripting rule the pattern-matching syntax does not require any changes when compared to an ordinary PWConstraints rule. The Lisp-code part, however, does not return a truth value. Instead it contains a list of Lisp expressions that perform various side effects to the score such as adding expressions. The approach described here has several advantages: 1. The syntax of the pattern-matching language is compact and powerful. 2. There is a built-in mechanism in MA-PMC to map through all the notational objects in an ENP score. There is no need for the user to write tedious loop constructs. 3. MA-PMC provides rich knowledge about the melodic, rhythmic and harmonic properties of the score. This information can be used in the scripts. ENP provides a set of specialized Lisp-functions that can be used to manipulate the appearance and content of ENP objects. Virtually every notational attribute can be accessed and manipulated with the scripts. Among the scripting properties of ENP are: color; channel and velocity; note-head type; enharmonic identity; time-signature, key-signature and tempo; start-time and duration; x-offset and y-offset, etc.

100 4 Some Scripting Examples In this section we examine some ENP scripts in detail. Due to space limitations the user-interface is not shown here. The ENP script-editor, however, contains a text editor to input the script code, a few options to choose from (e.g., if the script is applied to a selection only) and a list of the scripts already applied with the possibility for selective undo. 4.1 Adding an Articulation Pattern In our first example we use a script to insert a repeating articulation pattern to a passage of sixteenth notes. In this case the selected articulation pattern is slur-staccatostaccato. This particular example underlines the fundamental idea behind scripting: to perform a potentially laborious and repetitive task. We begin by defining the script: (*?1?2?3?4 ;1 (?if (when (downbeat??1) ;2 (add-expression slur?1?2) ;3 (add-expression staccato?3) ;4 (add-expression staccato?4)))) ;5 The first row of the script (1) shows the pattern-matching part containing a wild card and four named variables (?1,?2,?3 and?4). The second row (the beginning of the Lisp-code part), in turn, checks if the first of the four notes falls on a downbeat. Provided that it does, the next row (3) of the script adds a slur to the first two notes,?1 and?2, and the remaining two rows (4 and 5) add staccatos to the last two notes,?3 and?4, respectively. Figure 1 gives the output of the script. The benefits of this kind of a script are naturally more obvious when the musical passage in question is very long. In this case, the pattern-matching part (1) shows a wild card and a group of three named variables (?1,?2 and?3). The first row of the Lisp-code part (2) adds a special expression, called group 1, to the three notes. The set-class identity of the notes is supplied as a parameter for the group indicated by the keyword :info (3). Figure 2: The melodic excerpt is analyzed by the script and the analysis information is added to the score accordingly (Berg: Lyrical Suite). It is to be noted, however, that in this case the information added to the score is static. Should the user, for example, transpose any of the notes in Figure 2, the analysis information shown by the expressions would become incorrect. ENP provides dynamic expressions for this kind of analytical purposes, and they should be preferred over the static ones in practical applications. The next two examples show the advantage of dynamic expressions used along with scripting. 4.3 Adding Trumpet Fingerings There is sometimes the need to explicitly display the fingering information of different instruments in a score. This could be done for pedagogical purposes or for providing control information for a virtual instrument model (see for example Laurson et al. 2001) The script for adding trumpet fingerings for each note in the score is defined as follows: (*?1 (?if (add-expression trumpet-fingering?1))) Figure 1: An articulation pattern is applied to passage of music with the help of a script. 4.2 Adding Analysis Information The second script analyzes the given melodic line by calculating the set class identity of every group of three consecutive notes (see Figure 2). This example demonstrates the ability of the script to access the information contained by the notational objects. The script is defined as follows: (*?1?2?3 ;1 (?if (add-expression group?1?2?3 ;2 :info (set-class-name?1?2?3)))) ;3 Figure 3 shows the output of the script. This may seem trivial at first but the complexity of the case, however, becomes apparent after the user changes the musical information, for example, by transposing. In Figure 4 the user has transposed the musical material by a whole-step. This shows that the fingering expressions inserted by the script are dynamic, i.e., they adjust their visual representation according to the notational objects that they are attached to. There is no need to double check the notation or re-run the scripts in order to keep the score up to date. 1 A group is a specialized expression that displays any user definable textual information as a part of the music notation. The extent of the group is also visually indicated in the score.

101 If the beaming of the note has two levels (5) a grave accent is added (6). Figure 3: Trumpet fingerings inserted into the score by the script. Figure 4: Fingering expressions are dynamic, so when the musical material is transposed the fingerings are updated accordingly. 4.4 An Advanced Example We conclude with a relatively complex example. Here we use an excerpt from opus 25 by Arnold Schoenberg as a starting point. The aim is to fill a score with some of the articulations that are found in the original score with the help of a script. To make the example more straightforward we concentrate on two types of articulations, grave accents and slurs. All other attributes that are found in the original are omitted. In the original score there are always slurs between two notes if they form a large interval (the adjacent notes are more than an octave apart) provided that there is not a rest between them. Also, each sixteenth note has a grave accent. In this case we use the musical knowledge of PWConstraints and ENP to determine the type and position of the inserted articulations. This is done according to the following two principles: the size of the melodic movement or the beaming level of the notes. The result, which is shown in Figure 5, can be accomplished with the following script containing two scripting rules: (*?1?2 (?if (when (and ;1 (> (abs (- (m?2) (m?1))) 12) ;2 (not (prev-rest??2))) ;3 (add-expression slur?1?2)))) ;4 (*?1 (?if (when (= (beaming-level?1) 2) ;5 (add-expression accent-grave?1)))) ;6 The first row of the script shows a wild card and two named variables (?1 and?2). In the next two rows we check if the size of the melodic movement is greater than an octave (2) and also that there is no rest between the two notes (3). If both conditions are met we insert a slur expression (4). The latter part of the script is quite straightforward. Figure 5: The slurs and grave accents inserted by a script. The type and placement of the expressions has been determined by the script according to the duration of the notes or the size of the melodic movement. 5 Conclusions This paper demonstrated how ENP can be scripted using the PWConstraints pattern-matching language. The power of the pattern-matching language combined with the intelligent expression handling of ENP provides a powerful tool for different kinds of compositional and analytical applications. The scripting scheme presented here is useful when used simply as a speedy entry to attach, for example, articulations or articulation patterns. It can also be used to perform more sophisticated operations including music analysis and visualization. References Kuuskankare, M. and M. Laurson (2002, September). ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL. In Proceedings of International Computer Music Conference, Gothenburg, Sweden, pp Laurson, M. (1996). PATCHWORK: A Visual Programming Language and some Musical Applications. Studia musica no.6, Sibelius Academy, Helsinki. Laurson, M., C. Erkut, V. Välimäki, and M. Kuuskankare (2001, Fall). Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis. Computer Music Journal 25(3), Laurson, M. and M. Kuuskankare (2001). A Constraint Based Approach to Musical Textures and Instrumental Writing. In CP01 workshop on Musical Constraints, Cyprus. Leler, W. (1988). Constraint Programming Languages, Their Specification and Generation. Addison-Wesley Publishing Company. Woo, M., J. Neider, T. Davis, and D. Shreiner (1999). OpenGL Programming Guide (3rd ed.). Massachusetts, USA: Addison Wesley.

102

103 Publication [P4] [P4] M. Laurson and M. Kuuskankare. From RTM-notation to ENP-score-notation. In Journées d Informatique Musicale, Montbéliard, France,

104 From RTM-notation to ENP-score-notation Mikael Laurson 1 and Mika Kuuskankare 2 1 Center for Music and Technology, 2 Department of Doctoral Studies in Musical Performance and Research. Sibelius Academy, P.O.Box 86, Helsinki, Finland laurson@siba.fi, mkuuskan@siba.fi Abstract This paper discusses some recent developments within a compositional environment called PWGL. Our focus is to present how score information is represented in PWGL. We give some background information concerning the rhythmic notation that was used in PatchWork (a predecessor of PWGL). After this we show how this notation has been expanded so that it allows to generate very detailed scores that can contain besides the basic rhythmic structures also other information such as grace-notes, instrumentation, pitch and expressions. 1 Introduction PWGL (Laurson and Kuuskankare 2002) is a visual programming language based on Lisp, CLOS and OpenGL. PWGL is tool for computer assisted composition and sound synthesis. PWGL is based on similar concepts than PatchWork (PW, Laurson 1996) such as direct relation to its base languages Lisp and CLOS, musical objects and visual editors. The main difference between PW and PWGL is that the graphics part of the implementation is based now on OpenGL and not on QuickDraw as was the case with PW. OpenGL (Woo et al. 1999) has many attractive features and offers several advantages when compared to QuickDraw (for more details see Laurson and Kuuskankare 2002). In this text we will concentrate on the way complex musical scores are represented in PWGL. Scores are of primary importance in our system and they can be used in many compositional and analytical applications such as to produce musical material for instrumental music. Scores have been also used successfully to produce control information for physical models of acoustical instruments (see for instance Laurson et al and Välimäki et al. 2003). The Score-editor of PWGL utilizes internally the ENP2.0 music notation package. ENP2.0 (Kuuskankare and Laurson 2002) is based on the former ENP user library of PW (Kuuskankare and Laurson 2001). The Score-editor combines all PW music notation related editors (i.e. chord-editor, chord-sequence-editor, RTM-editor, PolyRTM-editor) into one package. ENP2.0 supports both mensural and non-mensural notation. Also ENP2.0 has been completely rewritten and is now based on OpenGL. The purpose of ENP2.0 is to provide professional notational capabilities, a graphical user interface and powerful object structures for compositional use. As PWGL, ENP2.0 is programmed with LISP and CLOS thus being extendible and open. It also provides full access to the musical structures behind the notation allowing ENP2.0 to be controlled algorithmically. Furthermore, ENP2.0 provides both standard and user definable expressions. Algorithmic control requires a high-level representation of the complex structural information behind the visual front-end of a musical score. To be useful this representation has to fulfil the following criterias. First, it has to be compact so that we are able to handle large scores efficiently. Second, it should be understandable to a human reader so that it could be used directly as input to build scores. Third, it should be easily translatable into object structures of the final score, and vice versa, any score should be translatable back to the highlevel representation. Finally, it should be extendible in order to meet future improvements and extensions of the current environment.

105 The RTM-package (Laurson 1996) of PW allowed to generate complex rhythms out of a list representation for beats. Henceforth we will call this list representation RTM-notation. In the current PWGL implementation the RTM-notation representation has been augmented in several ways. In the following we first introduce the PW RTM-notation and describe its most important features. After this we go over to the current implementation and show step by step how it can handle scores with increasing complexity. We discuss also one elaborate score example and end with some concluding remarks. 2 RTM-notation in PW The section gives a synopsis of the RTM-notation syntax used in PW to represent scores. We start with Lisp lists that are converted to metrical rhythms. Lisp lists is a natural choice as they allow easily to build hierarchical structures needed for our purposes. In the following we show the basic components of the system and after this discuss several examples that can be found in the western notation practice (Read 1980). 2.1 Beat-list The starting-point for a Lisp representation of a beat is a list. This list is called a beat-list and has two components: beat-count and rtm-list. Beat-count is a positive number, usually an integer. rtm-list, in turn, is a list of numbers. The numbers, called rtm-values, are usually positive integers, but can also be negative integers or floats. Let us assume the following example (Figure 1): Figure 1: Some basic rhythms with their beat-list representations. Figure 1 gives a measure containing four beats. Below each beat is the corresponding beatlist. Thus, for example, for the first quarter-note, the beat-list is (1 (1)). This detail is given also in Figure 2: Figure 2: A quarter-note and its beat-list representation. beat-count indicates how many units of the basic pulse is needed for a beat. The basic pulse, in turn, can be inferred from the denominator of the time signature. Thus, in the example above, the basic pulse is a quarter-note. As the beat-count is 1, the length of the beat in Figure 2 will be one quarter-note. The rtm-list consists of proportional values and has three properties: (1) the sum of all numbers in the rtm-list, (2) the length of the rtm-list and (3) the values of the distinct numbers. The sum gives the number of units into which the beat must be divided. The subunit of the beat equals 1 divided by the sum. Next, the length of the rtm-list determines the number of notes inside a beat. Finally, the rtm-value n indicates that n sub-units are to be assigned to the corresponding note. As the rtm-list in our example consists of only the number 1, we have only one note with a length of a quarter-note. Figure 3 gives the second beat of Figure 1, with the beat-list (1 (1 1)). The beat lasts for one quarter-note (beat-count = 1). As the sum of the rtm-list elements is 2, the sub-unit is 1/2. The beat contains two notes, both being 1/2 of the length of the beat, i.e. eighth-notes (1/4 * 1/2 = 1/8).

106 Figure 3: Two eighth-notes and the corresponding beat-list. In the third beat of the example in Figure 1 above, the beat-list is (1 (2 1)). The beat will last for one quarter-note (beat-count = 1). As the sum of the rtm-list is 3, the sub-unit is 1/3 of the length of the beat. The length of the rtm-list is 2. The length of the first note is 2/3 (2 * 1/3) of the beat, that of the second note 1/3 (1 * 1/3). Thus, the first note is a "sixth-note" (1/4 * 2/3), the second a "twelfth-note" (1/4 * 1/3). In the fourth beat of Figure 1 the rtm-list is (1 2), as opposed to the (2 1) of the previous beat. Consequently, the length of the first note will be 1/3 of the length of the beat, that of the second 2/ Simple Rhythms The examples below (Figures 4-6) show beat-list representations describing rhythms with different sub-units. The sub-units range from 1/4 to 1/6: Figure 4: sub-unit = 1/4. Figure 5: sub-unit = 1/5. Figure 6: sub-unit = 1/6.

107 Figure 7 gives an example with beat-count 2. Hence, each beat gets two units (2 * quarternote) of the basic pulse: Figure 7: beat-count = Embedded Rhythms Until now all the rtm-lists have been simple lists, that is, only one level deep. To be able to represent embedded rhythms, we add more levels to the rtm-list. In Figure 8, the top example starts with a simple beat-list (1 (1 1)). Then, in the right-hand side of the top example we replace the second 1 of the rtm-list with a beat-list (1 (1 1 1)). The result is a new embedded beat-list: (1 (1 (1 (1 1 1)))). The second example starts with a beat-list (2 (1 1 1)). We replace each 1 of the rtm-list with (1 ( )). Figure 8: The construction of embedded rhythms. 2.4 Compound Rhythms In compound rhythms the grouping of notes is controlled by the beat-count. In Figure 9 the time signature is 8/8 and the grouping is Figure 9: Compound rhythms. 2.5 Rests and Tied Notes Until now, all the rtm-list examples contained only positive integers. To represent rests and tied notes, we allow the numbers inside the rtm-list to be negative numbers or floats. If a number in a rtm-list is negative, it is considered to be a rest. If it is a float, it represents a tied note. The rhythm in Figure 10 shows a combination of note-heads, rests and tied notes. Figure 10: A combination of note-heads, rests and tied notes.

108 3 ENP-score-notation in PWGL In this section we present a new and enriched list notation called ENP-score-notation. We describe how the ENP-score-notation concept differs from the RTM-notation presented above. ENP-score-notation is based on some important concepts of RTM-notation, such as the hierarchical PW beat-list. For several reasons, this approach was adapted also in ENP. First, we wanted to ensure basic backwards compatibility between PW music editors and those of PWGL. Second, the list representation is uniform and proven to be powerful in practical use. And third, it was easy to extend the expressiveness of the syntax, as required by ENP, without scarifying the functionality or backwards compatibility. In ENP the musical structures of PW have been modified and extended to add new functionalities and hierarchies (Kuuskankare and Laurson 2001). Thus, also the ENP-scorenotation is enriched to reflect these changes. The starting point in the current implementation is to use the PW beat-list representation. Using the notation given above it is possible to create complex beat structures. To create higher-level structures (i.e., measures, voices, parts, scores) we collect lower-level structures as lists. For instance, beat-lists can be used to build measures. The measures, in turn, can be used to build even higher-level structures. 3.1 ENP Score Structure An ENP score is built out of hierarchical object structures. Thus a score consists of a list of parts, a part consists of a list of voices, a voice consists of a list of measures and finally a measure consists of a list of beats. For example we can create a score out of the following list representation that reflects directly the ENP score structure (note that both measures consists of beat-lists as described in Section 2). ( ; score ( ; part1 ( ; voice1 ((1 (1)) (1 (1 1 1)) (1 (1 2 1))) ; measure1 ((1 (1 2 2)) (1 (1 2)) (2 (-1 2 1))) ; measure2 ) ) ) Figure 11 shows in practice how the previous ENP-score-notation example is used in PWGL to build scores: Figure 11: A PWGL patch constructing scores. 3.2 Keyword Extensions The basic ENP-score-notation system described until now can be augmented by inserting keywords inside the list structure. This allows to define more precisely the resulting score. A typical example is for instance a measure that should have knowledge of its time signature in

109 order to work properly. If no information is given then default values are used. For instance in the example of Figure 11, the default denominator value of the time signature (or :low) is a quarter-note or 4 (the nominator value is calculated automatically using the beat-count values of the beat-lists), the default pitch value (or :midi) of a note is middle-c or 60, and so on. Typically the used keywords are slot-names of the resulting CLOS objects. This feature is useful as it allows to add new slots in the existing musical objects without having to modify the source code that takes care of the translation of the ENP-score-notation to an ENP score. Each level of the ENP score structure has its own set of keywords. For instance, following keywords can be used: :instrument, :staff, :clef-number, :low, :metronome-value, :start-time, :notes, :expressions, :midi, :velocity and :channel. Here are some ENP-score-notation examples dealing with various keywords accompanied with the resulting score (Figures 12-14): ((:instrument "Piano" :staff piano-staff ((:low 8 :metronome-value 72 (1 (1 1 1)) (1 (1 (1 :notes ((60 :clef-number 0) (60 :clef-number 1))))))))) Figure 12: A score with keywords for instrument, staff, time signature, clef-number and metronome. (((((4 ((1 :notes (72 64)) (1 :notes (72 65)) (1 :notes (72 67)) (1 :notes (71 65)))))) (((4 (2 (2 :notes (62)))))))) Figure 13: A two-voice part example with pitch information. (:Title "Duo" :spacing 0.7 (:instrument "Flute" (((1 ((1 :notes (72)))) (1 ((1 :notes (72) :expressions (:staccato)) (1 :notes (74) :expressions (:staccato)) (1 :notes (76) :expressions (:staccato)) (1 :notes (77) :expressions (:staccato)) (1 :notes (78) :expressions (:staccato))) 'grace-beat) (3 ((1 :notes (79) :expressions (:trillo))))))) (:instrument "Violoncello" :staff bass-staff (((1 ((1 :notes (36 43) :expressions (:accent :f)))) (3 ((1 :notes (52 60) :expressions (:accent))))))))

110 Figure 14: A two part example with instrument, staff, pitch and expression information. 3.3 Special Extensions The PWGL ENP-score-notation system allows also to give extra information that defines the type of a beat-object. One example of these special structures is the grace-beat. Grace-beats are used to define grace-chords or grace-notes in an ENP score. An example of this can be found in Figure 14 where the upper part contains an ascending grace-note sequence. Gracebeats are given by inserting at the end of the beat-list definition the symbol grace-beat. The following expression: (((((1 (1 1) 'grace-beat) (1 (1 1 1)) (1 ((1 (1 1)) (1 (1 1 1) 'grace-beat) (1 (1)))))))) gives as a result the score that can be seen in Figure 15: Figure 15: An example with two grace-beats. A similar extension allows to create accelerando- or ritardando-beats. The following example shows how to define these special beats (see also Figure 16): (:spacing 0.33 :zoom 2.0 :pan-y ((((1 ( ) 'accelerando-beat) (1 ( ) 'ritardando-beat))))) Figure 16: An example containing one accelerando-beat and one ritardando-beat. 3.4 Group-expressions ENP expressions can be applied to a single note or chord or to a group of notes or chords. We have already seen several examples of the former expression type. This subsection discusses the latter case which is called a group-expression. Group-expressions can overlap freely in the score. They are interesting for our study as typically they do not obey the strict metrical hierarchy imposed by the beat-list and measure notation. Thus we need a special syntax to generate them in a metric context. Group-expressions can be inserted to a chord using an appropriate keyword to define the class and instance identity of the expression in question. The keyword consists of two parts. It begins with an ENP class-name (e.g., 'group', 'slur', 'bpf', etc.). In addition of this, there can be an optional numeral that defines a group identity. The group identity numeral is used to distinguish between different instances of group-expressions. Thus, ':group1' would denote an

111 ENP object of class 'group' with identity value '1'. In this case, all subsequent references to ':group1' would point to the same instance. Furthermore, when an expression is introduced for the first time, any number of initialization-keywords can be supplied. In this case, the expression is given as a list consisting of the expression keyword and the required initialization-keyword/value pairs, for example: (:group1 :print-symbol "3-11a"). Below we find an example with three groupexpressions: (((((1 ((1 :notes (73) :expressions (:slur1)) (1 :notes (75) :expressions (:slur1 (:group1 :print-symbol "M1") )) (1 :notes (76) :expressions (:slur1 :group1)) (1 :notes (78) :expressions (:slur1 :group1 (:group2 :print-symbol "M2"))))) (1 ((1 :notes (74) :expressions (:slur1 :group1 :group2)) (1 :notes (76) :expressions (:slur1 :group1 :group2)) (1 :notes (77) :expressions (:slur1 :group1 :group2)) (1 :notes (79) :expressions (:slur1 :group1 :group2)))) (1 ((1 :notes ((75 :enharmonic 1)) :expressions (:slur1 :group1 :group2)) (1 :notes (77) :expressions (:slur1 :group2)) (1 :notes (79) :expressions (:slur1 :group2)) (1 :notes ((80 :enharmonic 1)) :expressions (:slur1 :group2)))))))) Figure 17: Group-expressions example containing a slur and two overlapping groups called 'M1' and' M2'. 4 Score example We end with a complex ENP-score-notation example which combines several features described in the previous subsections. The resulting score is found in Figure 18. (((((2 ((1 :notes (52) :expressions ((:slur1 :slope -6.0) :pp)))) (1 ((1.0 :notes (52) :expressions (:slur1)) (1 :notes ((58 :enharmonic 1)) :expressions (:slur1)) (1 :notes (59) :expressions (:slur1)) (1 :notes (62) :expressions (:slur1)))) (1 ((1 :notes ((63 :enharmonic 1)) :expressions (:slur1 :crescendo1)) (1 :notes (65) :expressions (:crescendo1)) (1 :notes ((66 :enharmonic 1)) :expressions (:crescendo1)) (1 :notes ((68 :enharmonic 1)) :expressions (:crescendo1)) (1 :notes (69) :expressions (:crescendo1)))) (1 ((1 :notes ((72)) :expressions (:slur2 :f)) (1 :notes ((73 :enharmonic 1)) :expressions (:slur2) :x-offset 1.0) (6 :notes (79) :expressions (:fp :crescendo2) :x-offset 2.0)))) ((1 ((1.0 :notes (79) :expressions (:crescendo2)) (-1 :notes (72) :expressions (:fz)))) (2 (-1 (1 :notes ((72 :note-head :x)) :expressions (:accent-grave :ff (:group1 :print-symbol "slap" :expression-fore-color 0))) (1 :notes ((69 :note-head :x)) :expressions (:accent-grave :group1)) (1 :notes ((73 :note-head :x)) :expressions (:accent-grave :group1)) (1 :notes ((68 :note-head :x)) :expressions (:accent-grave :group1)) (1 :notes ((66 :note-head :x)) :expressions (:accent-grave :group1)))) (1 ((1 :notes (77) :expressions (:glissando1 :fp :crescendo3)) (1 :notes ((80 :note-head :invisible)) :x-offset 1.0))) (1 ((1 :notes (83) :expressions (:glissando1 :fz :staccato :crescendo3))))))))

112 Figure 18: A complex example containing dynamics, single expressions, group-expressions and special note-heads. 5 Conclusions This paper gave an overview of some recent PWGL developments dealing with a novel score representation called ENP-score-notation. We first introduced the PW beat-list protocol which is used to construct metric structures. After this we augmented the beat-list scheme with keywords and special extensions in order to be able to build scores containing parts, voices, measures, chords and notes. The system allows also to define additional notational attributes such as expressions, grace-notes and group-expressions. References Kuuskankare M. and M. Laurson "ENP, Musical Notation Library based on Common Lisp and CLOS". In Proc. of ICMC 01, Havana, Cuba, pp Kuuskankare M. and M. Laurson "ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL". In Proc. of ICMC'02, Gothenburg, Sweden, pp Laurson M. and M. Kuuskankare PWGL: A Novel Visual Language based on Common Lisp, CLOS and OpenGL. In Proc. of ICMC'02, Gothenburg, Sweden, pp Laurson M., C. Erkut, V. Välimäki, and M. Kuuskankare "Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis." Computer Music Journal 25(3). Laurson, M PATCHWORK: A Visual Programming Language and Some Musical Applications. Doctoral dissertation, Sibelius Academy, Helsinki, Finland. Read, G Modern Rhythmic Notation. London. Victor Gollanz Ltd. Woo M., J. Neider, T. Davis, and D. Shreiner OpenGL Programming Guide. Addison Wesley, 3 rd edition, Massachusetts, USA. Välimäki V., Laurson M., and C. Erkut "Commuted Waveguide Synthesis of the Clavichord." To be published in Computer Music Journal.

113 Publication [P5] [P5] M. Kuuskankare and M. Laurson. Recent Developments in ENP-score-notation. In Sound and Music Computing 04, Paris, France,

114 RECENT DEVELOPMENTS IN ENP-SCORE-NOTATION Mika Kuuskankare DocMus Sibelius Academy Mikael Laurson CMT Sibelius Academy ABSTRACT In this paper we describe the recent developments on a text-based score representation format called ENP-scorenotation. ENP-score-notation is used to describe scores in Expressive Notation Package (ENP). We introduce the basic concepts behind ENP-score-notation and discuss several new features in detail along with some notational examples. We also examine some syntactic changes and point out the differences between the new and the old syntax. Furthermore, we propose a new interface to PWGL, including several new boxes, which allows to use ENPscore-notation to construct musical objects in a PWGL patch. 1. BACKGROUND Many different text-based formats have been developed for representing musical information. Some typical examples are Common Music Notation [10], GUIDO [3], LilyPond [9], and MusicXML [2]. There are also some Lisp-based representations that are able to preserve only a subset of the musical information, such as the rhythm. These include the RTM-notation of both PatchWork [6] and OpenMusic [1]. There are several advantages in text-based representation of a musical score: (1) the layout of notational objects is separated from the representation, (2) the musical typesetting is improved when the typesetting program improves, and (3) textual formats are human readable. ENP-score-notation [8] is a text-based format to represent ENP scores. ENP [4] is a LISP-based music notation program. It currently runs on MacIntosh OS X platform. The main purpose of ENP is to represent musical data for compositional and music analytical applications. ENPscore-notation is based on the hierarchical score structure of ENP. ENP-score-notation offers a large subset of definable attributes. The syntax is designed so that it also allows to add new features easily. The current subset, however, should be large enough for most practical purposes. The Lisp list format allows a straightforward way to generate and manipulate musical data. ENP-score-notation M.Kuuskankare and M.Laurson, Recent Developments in ENPscore-notation, in Sound and Music Computing 04, October can also be used in PWGL to represent musical objects. PWGL [7] is a multi-purpose visual programming environment specialized in music and sound related applications. PWGL uses ENP as its notational front end. The rest of the paper is organized as follows. Firstly, we briefly present the overall data structure of an ENP score. Secondly, we discuss ENP-score-notation in general. Next, we point out some syntactic changes in ENPscore-notation. Finally, we introduce a set of new PWGL boxes, relating to ENP-score-notation, along with some example patches. 2. ENP SCORE STRUCTURE IN BRIEF ENP scores are internally represented as hierarchical tree structures. For example, a score consists of a list of parts and parts, in turn, consist of list of voices, etc. In general, any higher-level structures are constructed by collecting lower-level structures as lists. Figure 1 gives an overview of the score structure. Figure 1. An example of an ENP score (above) and its object hierarchy displayed as a tree structure (below). On the beat-level ENP score representation is based on the concept of PW-beat which can be represented as a Lisp list. This list, called a beat-list, has two main components: beat-count and rtm-list. beat-count is a positive number, usually an integer. rtm-list, in turn, contains a list of numbers, called rtm-values. The type of an individual rtm-value has the following meanings: (1) positive integer represents a note, (2) negative integer represents a rest, and (3) floating-point number indicates a note that is tied to the previous one.

115 Figure 2 gives some typical examples of the beat-list notation along with their notational counterparts. More detailed explanation of PW-beat can be found in [6]. ENP score structure, in turn, is discussed in greater detail in [8]. maps exactly to the score hierarchy and is therefore easily transformable back to an ENP score. Below we give a simplified but complete example of the structure of ENP-score-notation along with the resulting score (Figure 3): Figure 2. Some simple rhythms demonstrating the use of rtm-values in the beat-list. 3. ENP-SCORE-NOTATION When we use a text-based format to describe the complex structural information contained by a musical score there are several things to be considered. The syntax has to be: (1) compact so that we are able to handle large scores efficiently, (2) understandable to a human reader so that it could be used directly as input to build scores, (3) easily translatable into the score representation (and vice versa), and (4) extensible in order to meet future extensions and improvements ENP-score-notation Syntax The structure of ENP-score-notation [8] reflects the hierarchical structure of the score. The syntax is similar to the LISP list syntax and also resembles CLOS in the way the optional keyword arguments are given. The basic syntax of ENP-score-notation describes only the rhythmic skeleton along with the overall structure of the score (i.e., parts, voices, measures, etc.). All other information (including pitches) is defined as optional keyword arguments. For example, in its simplest form, a triplet is notated as (1 (1 1 1)) 1. The pitch information 2 can be added by using a keyword :notes, e.g., (1 ((1 :notes ( )) (1 :notes ( )) (1 :notes ( )))). The :notes keyword refers to an actual slot in an ENP chord object. There are several advantages in this approach: (1) Since all the attribute names refer to actual slots in CLOS objects, features can be added in most cases without any additional coding. (2) Syntax is uniform and simple: everything else, besides rhythm, is defined by adding optional attributes. (3) Unless a particular attribute is given it is set to some reasonable default. (4) The hierarchical relations of notational objects can be determined by examining the textual representation. (5) The format 1 Even if not explicitly indicated the resulting triplet would in this case also contain some pitch information. The default pitch, unless stated otherwise, is c 1 (= middle c). 2 Pitches in ENP are generally represented as midi values, 60 being c 1, i.e., middle C. However, ENP can also represent microtones. They are notated by adding the relevant decimal fraction to the midi value. Thus, 60.5 would be c 1 raised by 1/4 (half a semitone). Figure 3. A simplified ENP-score-notation example (left) and the resulting score (right). One of the problems with strict hierarchical representation is how to express entities that do not obey the hierarchy. ENP-expressions are a typical group of objects that break the otherwise hierarchical representation described above. In the next subsection we briefly describe how ENP-expressions are represented in ENP-score-notation Syntax of ENP-expressions Currently ENP-expressions [4, 5] can be attached to either single chords or notes (single-expressions) or to groups of chords or notes (group-expressions). Expressions can be incorporated into ENP-score-notation by using a keyword :expressions followed by a list of expressions and possible initialization options. Expressions are identified by using an appropriate keyword indicating the class of the expression. Among valid expression keywords are, for example, :group, :slur, :bpf, :accent, :staccato, etc. In case of group expressions the keywords usually contain an additional numeral that indicates the instance identity of the expression. The instance identity numeral is used to distinguish between different instances of group-expressions (in case of overlapping slurs, for example). Thus, :slur1 indicates an ENP expression of class slur with the identity value 1. All subsequent references to :slur1 point to the same instance. Furthermore, when an expression is introduced for the first time, any number of initialization options can be supplied. This information consists of keyword and value pairs defining any additional properties of the expression. The components of an expression keyword are as follows: expression class : slur 1 instance identity numeral 4. CHANGES IN ENP-SCORE-NOTATION SYNTAX In this Section we enumerate some syntactic changes in ENP-score-notation and also present some new functionalities, such as the possibility to describe non-mensural notation.

116 4.1. Changes in Beat Types Formerly, the type of the resulting beat was indicated by inserting an extra symbol as the last element in the beat-list. For example, a grace-beat was created by using the symbol grace-beat, e.g., (1 (1) grace-beat). The new syntax, however, unifies the use of these kinds of optional attributes. When using the current version of ENPscore-notation the grace-beat is defined as follows: (1 (1) :class :grace-beat). Notice the inclusion of a keyword :class. The beat list can also have several other optional attributes. The following values for the :kind keyword are currently supported: :accelerando, :ritardando, and :feathered. The distinction between :class and :kind keywords also allow to mix these attributes, e.g., to create a grace-beat with accelerando beaming. Figures 4 demonstrates the use of the :kind keyword and Figure 5, in turn, gives an example of a grace-beat created with the keyword :class. (((((1 ( ) :KIND :ACCELERANDO))))) Figure 4. A beat with accelerando beaming. (((((1 ((1 (1 1) :CLASS :GRACE-BEAT) (-1))))))) Figure 5. A grace-beat 4.2. Non-mensural notation The current version of ENP-score-notation allows to represent non-mensural notation. The syntax is otherwise identical to that of mensural-notation, but instead of using proportional beat-counts and rtmvalues, non-mensural notation uses absolute start-times and relative offset-times. Thus a mensurally notated chord, (1 ((1 :notes ( )))), could potentially be translated to non-mensural notation as (0.0 :notes ( ) :duration 1.0), where the first floating point number represents the start-time of the chord. 3 Each note can also have an independent offset-time relative to the start-time of the chord. Next we give a complete example of non-mensural notation and the resulting score (Figure 6): 3 We assume that, in the mensural case, the time-signature is 4/4 and tempo is 60 beats/minute. Thus, the duration of one quarter-note is 1 second. (((((0.2 :NOTES ((60 :OFFSET-TIME -0.15) (66 :OFFSET-TIME 0.0) (71 :OFFSET-TIME 0.15)) :EXPRESSIONS ((:crescendo/1 :initial-dynamics :p :end-dynamics :f))) (0.5 :NOTES (77) :EXPRESSIONS (:crescendo/1)))))) Figure 6. A non-mensural score described with ENPscore-notation Changes in ENP-expression Syntax When using the old ENP-score-notation syntax the instance identity numeral is attached directly to the end of the expression keyword, i.e., :bpf1. This creates a potential conflict when used with expressions whose names contain a numeral, such as the string number expressions (:string1, :string2, etc.). The new syntax allows to separate the expression name and the instance identity numeral more precisely by using a slash (/). Now different instances of, for example, string number expressions can be denoted as :string1/1 and :string1/2. This approach clarifies ENP-score-notation syntax. The old syntax, however, is still valid and can be used alongside the new one Score-BPF notation The Score-BPF [5] is multipurpose graphical object that can represent breakpoint functions as a part of a musical texture. breakpoint functions, in turn, are piece-wise linear functions. It can contain an arbitrary number of breakpoint functions which, in turn, contain a list of points with x and y values. The x values are timing values, in seconds, relative to the start-time of a Score-BPF in the score. The new version of ENP-score-notation supports Score-BPF expressions. Currently there are two ways to define a point in a breakpoint function: (1) By using absolute values. In this case a point can simply be indicated by a list of two numbers, e.g., ( ). The first number in the list defines the position of the point in time relative to the starttime of the Score-BPF. The second value is a y value that should fall inside the user definable height of the Score-BPF. (2) By using values that are proportional to the length (in time) and height of the Score-BPF. The point is written as a list of two numbers ranging form 0.0 to 1.0. In this case it is required to include an optional keyword argument, :scaled t, indicating that the incoming values are to be scaled inside the Score-BPF (Figure 7). Moreover, the user can define a number of others attributes, such as type, name and height of the Score-BPF, the name and color of the breakpoint function, etc.

117 (((((1 ((1 :notes (72) :expressions ((:bpf1 :points ((( ) ( ) :scaled t :name "acc."))))) (1 :notes (74) :expressions (:bpf1)) (1 :notes (76) :expressions (:bpf1)) (1 :notes (77) :expressions (:bpf1)) (1 :notes (78) :expressions (:bpf1)))))))) enp-constructor -box for a beat object. The box is extended to add a class attribute (i.e., :class and :gracebeat). The lowest row of the box shows the corresponding input-box pair. Figure 7. A Score-BPF expression (:bpf1) inserted in the score with the help of ENP-score-notation. Notice also the use of the extra arguments :name and :scaled. 5. PWGL INTERFACE TO ENP-SCORE-NOTATION In this Section we present the interface between ENPscore-notation and PWGL. Due to its list based representation ENP-score-notation provides a natural way to construct musical structures in a PWGL patch. We introduce next three new boxes that allow the user to generate and manipulate ENP objects in a PWGL patch. Two of the boxes, enp-constructor and enp-object-composer, are used to convert ENPscore-notation to ENP objects. The third complementary box, enp-score-notation, is in turn used to convert an ENP score to ENP-score-notation enp-constructor -box enp-constructor -box is used to create a single kind of ENP object. The main-box consists of two required input-boxes: (1) Selector-input-box which is used to select the type of the resulting object. The box dynamically changes the number and type of required input-boxes as the user changes this value. (2) One or two value-inputboxes that accept as input either ENP-score-notation expression or ENP objects. (Figure 8) Any additional properties can be defined by extending the enp-constructor -box. Extending reveals one special input-box pair at the time. The collection of possible input-box pairs depends on the type of the resulting object. The input-box pairs, in turn, contain two inputboxes arranged in a row. The first input-box, selectorinput-box, is used to select an attribute keyword. A change in the selector-input-box performs a side effect to the second input-box, called state-input-box. The state-input-box dynamically changes its type according to the value of the selector-input-box. If the value of the selector-inputbox is, for example, :start-time, the state-input-box is of type float-input-box. On the other hand, if the value of the selector-input-box is :expressions, the state-input-box is an expression menu-input-box containing all the possible ENP-expressions. Figure 8 shows an Figure 8. enp-constructor -box is used to create a grace-beat enp-object-composer -box enp-object-composer -box can be used to convert any lower level object to any of the higher level objects (see the score hierarchy in Section 2). Thus, a note could potentially be converted to a score, etc. enp-object-composer -box has several useful properties: (1) it allows fast visualization of any musical object (e.g., convert a note directly to a score), (2) it can handle either single objects or lists of objects, (3) it can be used as a shorthand to create any higher level objects out of any combination of lower level objects (types can be freely mixed), and (4) the input format is similar to the hierarchical score representation and can be used to further modify the structure of the resulting object. Figure 8 shows one application to enp-object-composer -box. In this case a grace-beat is converted directly into a score. Let us next examine a more complex patch given in Figure 9 (see Appendix). The main idea behind this patch is to demonstrate how to create different score structures from the same input data just by adding one extra parenthesis each time. The original input in this case is a list of two notes. In the first case (a) this input creates a score with two parts each containing one note. In (b) one extra parenthesis is added around the two notes. This, in turn, creates a score with one part containing two voices. Each voice again contains one note (the stems of the notes are drawn in different directions). In (c) two extra parenthesis are added around the the notes and the resulting score contains a part with two measures. In the last case (d) there is a total of four parenthesis around the notes. The resulting score contains one part with a measure containing two notes. This procedure can be continued to create even deeper structures.

118 5.3. enp-score-notation -box enp-score-notation -box is used to convert a score to ENP-score-notation expression. Unless stated otherwise, this box collects all the optional attributes for each object in the score. 4 It is, however, possible to have control over which attributes are collected. The enp-score-notation -box contains a menu-inputbox and a dialog-input-box for this purpose (see Figure 10). The menu-input-box can be used to choose between :exclude and :include keywords. The dialog-inputbox, in turn, can be used to select the set of attributes from a dialog containing all the possible attributes. According to the value of the menu-input-box these attributes are then either included in or excluded from the result. Figure 10, in the Appendix, gives an example of the extreme effects of :exclude and :include. The topmost score is the starting point containing a short passage of music with some expression markings and special note-heads. The enp-score-notation -box on the left (a) has the :exclude parameter set to NIL meaning that nothing is excluded from the result. This creates an exact copy of the original score (b). The enp-score-notation -box on the right (c) has the :include parameter set to NIL. In this case none of the optional attributes is included. Only the rhythmic skeleton is collected as shown in (d). 6. CONCLUSIONS This paper presented some recent developments in ENPscore-notation. Several new features were introduced. Also some syntactic changes were proposed that further clarified or extended the functionality of ENP-scorenotation. Furthermore, a set of dedicated PWGL boxes were introduced. These boxes provide a bridge between ENP-score-notation and PWGL and allow an algorithmic control over ENP-score-notation. There is still some room for improvement. The two ways to describe breakpoint functions, presented in this paper, are quite primitive. There is no apparent relation between the points and the notational objects (except for the start and end points). The syntax should be expanded so that it would allow the notational objects to be referenced by a symbol or an index. 4 There is a relating mechanism in ENP called automatic source code generator. The purpose of the automatic source code generator is to build a Lisp representation of a CLOS object. It can then be used to re-create an exact copy of the object. However, in order to optimize the Lisp representation, the automatic source code generator also compares the properties of the CLOS objects to their default values and determines if the value of the particular property needs to be included. This mechanism is quite sophisticated and can also handle arbitrary deep hierarchical structures. The same kind of strategy is used in the case of ENP-score-notation. The enp-score-notation -box actually collects only those attributes that are different from their default value. For example, let us say that there is a note whose duration is 1.0 seconds. This is also the default duration of a note object. In this case the :duration keyword and the corresponding value are not included in the ENP-score-notation expression. The current inclusion functionality of enp-score-notation -box is somewhat ambiguous. As explained in the footnote 4 the inclusion of a certain attribute is dependent not only on the choices made by the user but also the underlying decisions made by the system. There should be a mechanism to force certain attributes to be present in the ENP-score-notation expression (e.g., force-include feature). This would ensure that the needed attributes would always be collected. The information could then be used to construct other objects that explicitly rely on the existence of such data. Instead, the current implementation requires that the user has knowledge about the default values of the various attributes. 7. ACKNOWLEDGMENTS The work of Mikael Laurson has been financed by the Academy of Finland. 8. REFERENCES [1] Gerard Assayag, Camillo Rueda, Mikael Laurson, Carlos Agon, and Olivier Delerue. Computer Assisted Composition at IRCAM: From PatchWork to OpenMusic. Computer Music Journal, 23(3):59 72, Fall [2] M. Good and G. Actor. Using MusicXML for File Interchange. In Third International Conference on WEB Delivering of Music, page 153, Los Alamitos, CA, September IEEE Press. [3] H. H. Hoos, K. A. Hamel, K. Renz, and J. Kilian. The GUIDO Music Notation Format - A Novel Approach for Adequately Representing Score-level Music. In Proceedings of International Computer Music Conference, pages , San Francisco, [4] Mika Kuuskankare and Mikael Laurson. ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, September [5] Mika Kuuskankare and Mikael Laurson. ENP- Expressions, Score-BPF as a Case Study. In Proceedings of International Computer Music Conference, pages , Singapore, [6] Mikael Laurson. PATCHWORK: A Visual Programming Language and some Musical Applications. Studia musica no.6, Sibelius Academy, Helsinki, [7] Mikael Laurson and Mika Kuuskankare. PWGL: A Novel Visual Language based on Common Lisp, CLOS and OpenGL. In Proceedings of International Computer Music Conference, pages , Gothenburg, Sweden, September [8] Mikael Laurson and Mika Kuuskankare. From RTM-notation to ENP-score-notation. In Journées

119 d Informatique Musicale, Montbéliard, France, [9] Han-Wen Nienhuys and Jan Nieuwenhuizen. Lily- Pond, a system for automated music engraving. In XIV Colloquium on Musical Informatics (XIV CIM 2003), Firenze, Italy, May [10] Bill Schottstaedt. Comon Music Notation. In Beyond MIDI, The Handbook of Musical Codes. MIT Press, Cambridge, Massachusetts, 1997.

120 Appendix Figure 9. An Example showing the behavior of the enp-constructor -box. By manipulating the input data, different kinds of score structures can be created. Figure 10. An example showing the two extremes of enp-score-notation -box. On the left: an exact copy is made. On the right: only the rhythmic skeleton is retained.

121 Publication [P6] [P6] M. Kuuskankare and M. Laurson. ENP Expression Designer - a Visual Tool for Creating User Definable Expressions. In International Computer Music Conference, pages , Barcelona, Spain,

122 ENP EXPRESSION DESIGNER A VISUAL TOOL FOR CREATING USER DEFINABLE EXPRESSIONS Mika Kuuskankare Sibelius Academy DocMus mkuuskan@siba.fi Mikael Laurson Sibelius Academy CMT laurson@siba.fi ABSTRACT In the present paper we introduce a new tool called ENP Expression Designer (ED). It is aimed at visually creating ENP-expressions. Currently, ENP allows to create new expressions by using Lisp and CLOS. The strictly textual interface can, however, be demanding to those users that have limited or no experience in programming. ED provides an easy and straightforward way to define new expressions through a graphical user interface. Through inheritance the ED allows to reuse the built-in properties of the existing ENP-expressions. We also provide a set of graphical primitives for drawing lines, text, etc. Additionally, more experienced users can take full advantage of the OpenGL graphical language. 1. INTRODUCTION Especially in modern music notation there is the need for special expression markings. There is an increasing amount of instrument specific playing techniques and new ones are invented all the time. These markings can also be invented even piece per piece basis. This makes it difficult for a music notation program to provide a comprehensive collection of expression markings. Thus, it is more reasonable to provide an interface for the user that (s)he can use to build custom expressions. Two of the most widely used commercial notation programs, Finale [1] and Sibelius [2], provide the possibility to create user defined expressions. In Finale it is possible to create either text expressions or graphic expressions (shapes in Finale terminology). In Sibelius, in turn, the former group of expressions is called Symbols and the latter one Lines. Both programs provide tools to create both types of expressions. However, when using these tools it is usually not allowed to view the expressions in their natural context music notation. ENP [3] is a music notation program that has been developed in order to meet the requirements of computer aided composition, music analysis and virtual instrument control. It s purpose is to display scores using the common western notation. Furthermore, ENP provides a rich set of both standard and user definable notational attributes, called ENP-expressions. ENP is programmed in Common Lisp (LispWorks by Xanalys) and uses the OpenGL API to produce the notational graphics. ED is a visual tool that can be used to create custom expressions in ENP. ED provides a simple protocol that helps to define the basic properties of the user definable expressions. It also allows to define the visual appearance using Lisp. There are several advantages in the approach presented in this paper. One advantage that ENP provides is that the defined expressions can contain user definable data. This means that the data can be read and written both through the graphical user interface or algorithmically. The data can then be used in, for example, compositional or music analytical applications. Second, the expressions can be dynamic since they are defined using standard Common Lisp. Third, in ED the expressions are displayed in a fully functional ENP score, thus the user can observe how the expressions behave, for example, when the music is transposed or when the layout of the score is changed. This approach is different form any of the existing ones. For example, in Finale, the user definable expressions (shapes) are designed without the presence of any musical context. The rest of the paper is structured follows: first, we give a brief overview of ENP-expressions. Next, we introduce the ENP Expression Designer and give a few practical examples of its use. We end the paper with some conclusions and suggestions for future work. 2. OVERVIEW OF ENP-EXPRESSIONS ENP-Expressions are divided into two main categories: (1) standard expressions, including articulations and dynamics, and (2) non-standard expressions which include, for example, groups and Score-BPF s [3]. Every expression is attached either to a single notational object (single expressions) or to a group of notational objects (group expressions). ENP provides a collection of standard tempo-, dynamics- and articulation-symbols. Furthermore, it includes a set of non-standard notational attributes. New expressions can be created through inheritance. Next, we present some of the basic components of ENP-Expressions The Basic Components of ENP-Expressions All ENP-expressions share some common properties. These include, for example, print symbol, default print

123 position, user definable offset, and font scaler. While this is not a complete list these properties already define a bulk of the visual appearance, placement, and behavior of the expressions. The specific functions of each of the properties are further discussed in subsections 3.2 and 3.3. Figure 1 shows these properties in a musical context. Figure 1. The components of an ENP-expression. On the left the graphical representation of an accent. On the right, some common attributes to all ENP-expressions. In the class view the user must specify a unique class name and a superclass. By default all the user defined expressions inherit from the built-in expression -class. This is the fundamental class that provides the necessary data structures and behavior of all ENP-expression Properties view The properties view provides the following options: (1) Menu-category. All user-defined expressions are automatically added in the appropriate ENP menus where they are, in turn, arranged in groups according to the type and/or purpose. Here the user can either choose one of the existing menu categories or create a new one by providing a new category name. (2) Print symbol defines the string or the character which is used as the printed representation of the expression. (3) Print position. There are several predefined print positions which, in turn, are used to determine the default print position. The current choices are: :on-stem, :above-staff, :below-staff, :above-chord, and :notehead. The individual effects of these values can be seen in Figure ENP EXPRESSION DESIGNER ED is a visual tool for rapid expression designing and prototyping. The workflow typically consists of three steps: (1) defining class and inheritance, (2) defining properties, and (3) defining the visual appearance (coding). All these steps can be executed in a single ED window. The ED Window, in turn, contains the following components: class view, properties view, code view, and preview score (an example ED window is shown in Figure 3). Expressions are defined by first entering a unique class name and its superclass. Next, some additional properties are defined. These include among other the font typeface. Finally, the graphical outlook of the expression is defined by entering some Lisp code in the code view. The visual appearance of the new expression is shown in the preview score at the bottom of the ED Window. There are also buttons for displaying the preview and generating the expression definition in Lisp. The preview -button refreshes the preview score to give the updated visual representation of the expression. The print source -button, in turn, generates a text document containing the Lisp code needed to store and restore the user defined expression. Change in any of the aforementioned properties (except change in the Lisp code in the code view) provides synchronized feedback in the preview score. In case of the code view the user is required to press the preview - button to update the contents of the preview score. In the following Subsections we examine in detail the components of the ED Window Class view Figure 2. Five accent expressions attached to the chord. Each accent is given a different print position. The effect can be seen in the example along with the respective print position keyword. (4) Font can currently be either :times or :sonata. There are also some check boxes in the properties view. These can be used to determine whether the expression is standard notation, and whether the expression is saved along with the score or not Code view In the code view the user can write standard Lisp code (see Figure 3). The code defines how the expression is drawn in the score. The users can utilize any of the OpenGL functions provided by LispWorks. There are also some special graphical primitives provided by ENP that can be used to draw, for example, lines, polygons, circles, etc. The following list gives the complete set of graphical primitives currently available: draw-2darrow, draw-2d-box, draw-2d-bracket, draw-2d-circle, draw-2d-line, draw-2d-lines, draw-2d-point, draw-2dpolygon, draw-2d-quads, draw-2d-shape, draw-2d-text, draw-2d-texture, and draw-2d-triangles. 1 1 The list of the graphical primitives is quite self explanatory and it is not further discussed in this paper.

124 There are also some special variables that can be referred to in the code. The following variables contain some useful properties of the expressions: (1) x and y give the absolute position of the expression. The are calculated according to the position of the associated notation object and other constraints. These values are an aggregate of the default position (calculated according to the print position) and the user definable offset discussed above (see Subsection 3.2). (2) width gives the horizontal distance from the current notation object to the next one. (3) print-symbol contains the string or character defined by the user in the properties view. (4) font gives the font name as a keyword. Currently, the value of this variable is either :times (for readable text) or :sonata (for music notation). (5) instrument variable contains an ENP instrument object that, in turn, contains information about the name, range, etc. (6) expression is the current ENP expression object that is being drawn. This object can also be queried about extra information such as any user definable data stored in the expression. (7) self, in this case, refers to the notational object the expression is associated with (i.e., a note or a chord). Different properties of the notational object, including pitch, start-time and duration, can also be used in the expression code. Lisp is beneficial to thoroughly understanding the provided code examples A Percussion Mallet Symbol: Wirebrush As our first exercise we create a simple percussion mallet symbol denoting a wirebrush (see Figure 3). We begin the expression definition by providing a unique class name, wirebrush (1). In this case there is no need for any additional inheritance other than the basic expression behavior. Thus the default value ( expression ) displayed in the inheritance menu is sufficient for our purposes (2). The menu category is chosen to be User (3). Print symbol and font typeface, in this example, do not have any use since the expression in question is purely graphical (4). In music notation these kinds of instructions are usually written above the staff [4] and the print position is selected accordingly (5). In code view there are three lines containing some drawing instructions (6). Finally, the graphical representation of the expression can be seen in the preview score (7). The second note in this example is added to demonstrate how the user-defined expression behaves when the associated notational object is transposed. Note, how the expression avoids colliding with music notation Preview score The preview score displays the user defined expression as it would appear in a regular ENP score. The preview score is also fully editable so the user can observe how the expression behaves, for example, when the music is transposed, etc. The score can also be panned and zoomed freely so that the expressions can be examined and aligned in great detail. There is one additional feature to this view. As described above, the default expression position is calculated according to the selected print position and an additional user definable offset. This offset can be adjusted in the preview score by dragging the expression with the mouse. 4. SOME EXPRESSION DESIGNER EXAMPLES In this Section we give two examples on how the ED can be used to create new expressions. In the first example, wirebrush, we build a simple percussion mallet symbol [5]. This example demonstrates the use of some of the graphical primitives. In the second example, in turn, we create an expression that uses the knowledge about the spacing of the associated notational objects to fill a given space in the score. Due to space limitations the examples are given without any detailed explanation. The key points, however, are discussed along each example. Some knowledge about Figure 3. The wirebrush mallet symbol created with ED A Playing Technique: Scrape The second ED example focuses on a graphic expression that can be used to denote scraping an instrument [5]. As we draw the scraping symbol in the score it is required have knowledge about the length or extent of the associated notational object (Figure 4). This information is provided with the special variable called width. Next, we briefly describe the steps needed to define the scraping expression. The class name and inheritance are as shown in (1) and the menu category is again chosen to be User (2). In this case, it is required to define the print symbol and font typeface since the expression contains some written instructions along with the graphical information (3). The expression print position is above-staff

125 Figure 4. A playing technique expression created with ED. as indicated in (4). The lisp code part is once again offered without any detailed explanation (5). Suffice to say that the x and y -shapes are first defined and then drawn with a graphics primitive called draw-2d-shape. Additionally, the user defined print symbol is also drawn. Finally, the preview score (6), demonstrates how this particular expression reacts to the duration of the associated notation object due to the width -parameter (see code view). 5. CONCLUSIONS AND FUTURE WORK The current version of ED is already quite functional and can be used to quickly prototype and create new expressions. Some of the benefits in this approach are that it does not require extensive programming experience and the limited set of graphics primitives is quite easy to learn. It allows to create new expressions without the need to reproduce the low-level lisp code for each one. The number of lines of code the user is required to write is reduced to a fraction compared to the entirely text based approach. Furthermore, the synchronized visual feedback offered by ED is a valuable property as it allows the user to design and view the expressions in their native context. There are, however, several additional features and enhancements planned to make ED more useful as a general purpose tool: First, it is currently not possible to create group expressions. Group expressions require more options and possibly a dynamic property view. There are also a handful of additional methods that are user definable which in turn require more code view type of inputs. A dedicated group expression tab would be one solution to lay out the ED Window to facilitate this feature. Second, it should also be possible to use graphical tools for drawing the expression definition. This feature would require a graphics canvas and a set of graphical tools, such as lines, polygons and circles. It would also bring the ED closer to the more traditional tools offered by, for example, Finale. However, allowing to combine both algorithmic and hand drawn representations would make this approach more flexible. Third, there is currently no means to add slots to the user defined expression classes through ED. This is of primary importance when designing more complex objects that, for example, have different kinds of graphical representations depending on attributes selected by the user. Finally, although ED currently creates the menus for inserting the expression in the music notation there should be the possibility in ED to define the appropriate context sensitive menus for the expression itself. This would require a dedicated menu editor to be embedded in ED. The ED project was launched only recently and as can be seen in the missing features list above there is still quite a lot of work to be done. The current version is more of a proof of concept at this time. It requires some redesigning and additional coding to integrate all the listed features in ED. However, the fully mature application should prove to be beneficial not only for casual users but also for the more experienced ones. 6. ACKNOWLEDGMENTS The work of Mikael Laurson has been supported by the Academy of Finland (SA ). 7. REFERENCES [1] Finale User Manual. [2] Sibelius3 User Guide. [3] Kuuskankare, M. and M. Laurson. ENP2.0 A Music Notation Program Implemented in Common Lisp and OpenGL, Proceedings of the International Computer Music Conference, Gothenburg, Sweden, [4] Read, G. Music Notation. Victor Gollancz Ltd., [5] Risatti, H. New Music Vocabulary. A Guide to Notational Signs for Contemporary Music. Univ. of Illinois Press, Urbana, 1973.

126

127 Publication [P7] [P7] M. Kuuskankare and M. Laurson. ENP Note Head Designer. In Journées d Informatique Musicale, Paris, France,

128 ENP NOTE HEAD DESIGNER Mika Kuuskankare Department of Doctoral Studies in Musical Performance and Research Sibelius Academy Finland Mikael Laurson Centre for Music and Technology Sibelius Academy Finland ABSTRACT This paper presents a new visual editor called ENP Note Head Designer (henceforward ND). ND is aimed at assisting both novice and expert users to design custom note heads in ENP. Both visual and text-based interfaces are provided. Using the latter approach it is possible to take advantage of the power of Common Lisp, CLOS, and OpenGL (the base languages of ENP). The former approach, in turn, allows to use a set of graphical tools to define the note heads visual appearance. It is also possible to mix these approaches. Furthermore, the note heads are viewed and designed in an actual musical context. These concepts make the current system unique compared to the existing approaches. 1. BACKGROUND ENP Note Head Designer (ND) is part of an increasing collection of visual tools inside ENP [3]. Currently, these tools include (1) Color Editor, (2) Expression Designer, and (3) Note Head Designer. The purpose of these tools is to provide a visual interface for constructing complex music notation related CLOS objects and to provide assistance to those users that do not possess the needed programming skills to use the strictly text-based approach. It is important to realize, however, that the visual approach is not exclusively for the unexperienced users. Some objects may be realized more easily by using a purely graphical approach while the others may be more suitable to be defined using a programming language. Thus, a text-based approach is provided along with the graphical one. Uniform design principles have been applied to all aforementioned tools. The tools share same kinds of editors, components and functionality which makes them easier to learn. Most editors also contain an interactive feedback view. This makes it possible to design objects in their native environment, e.g, the designer sees the note head as it would appear in the score. Furthermore, visual synchronized feedback is one of the most important design concepts behind ENP. Every applicable GUI operation behaves this way and thus all ENP tools share the same design principles. The use of visual editors in our case has several advantages. First, the underlying CLOS syntax can be hidden to allow the user to concentrate on the task in hand (e.g., designing note heads). Second, following the purely textbased scheme involves routinely repeating several mandatory low level program components (i.e., class definitions, specialized method definitions, etc.). This kind of repetitive work load is considerably reduced. Finally, this approach is also less prone to errors since the syntax and most of the underlying CLOS code is handled automatically. Most of the existing commercial notation programs, such as Finale[1] or Sibelius[8], provide some kind of interface for the user to change the appearance of the note heads. The most fundamental and the most frequent approach is that the user can select a note head character and a font typeface. On the other hand, Finale additionally provides some graphical tools allowing to define the note head appearance by drawing arbitrary shapes in a graphics canvas. However, in this case the note heads are designed without the presence of any musical context. In Common Music Notation (CMN, [11]) it is also possible to define note heads algorithmically by using Common Lisp. Finally, LilyPond [9] allows to change the note head types algorithmically, by using Scheme programming language. ND is a visual tool that allows to design and view userdefinable note heads in ENP. ND provides an approach that mixes both the power of textual programming and the use of traditional graphical tools. All the needed program components and user-interface components (e.g., menus) are prepared automatically by ND. The note heads are also viewed and designed in a musical context. Moreover, the designed note heads are ready to be used in the current ENP session and can be saved in a file to be loaded either automatically or by demand. In the following sections we first describe the note head scheme used in ENP. Then we present the ND window and its components. After this we examine in detail the programming interface. Finally, we give a comprehensive set of examples prepared with the help of ND. The paper ends with some concluding remarks. 2. THE ENP NOTE HEAD SCHEME In this Section we describe the internal representation of note heads in ENP. We enumerate the different types of note heads and also give examples of their visual counterparts. Internally the note heads can be represented in the

129 following ways: (1) Strings or characters. In this case the object is usually the literal representation of the note head, i.e., the string or character is drawn as is using the default music notation font. The set of standard ENP note heads are defined in this way. Figure 1 gives a set of some character-based note heads in ENP. Figure 1. Character-based note heads in ENP. (2) Keywords act as symbolic references to specific CLOS methods that, in turn, draw the note heads according to the provided code. This approach is suitable for more complex and usually, but not necessarily, static note heads, which cannot usually be represented with a simple character or string. Figure 2 gives an example of one such note head, denoting a tambura effect used in guitar music. Figure 2. A tambura note head. (3) Objects. When note heads are expressed as objects every note head can have a different visual appearance and an individual set of data and properties. This allows, for example, to use different kinds of user editable shapes as note heads. The internal bpf note head is an example of this kind of an object (see Figure 3). In this case the note head is represented as a break-point function. It can be used for several different purposes, among others, to define the visual appearance of the note head itself by providing the note head shape or allowing the user to define one, or to contain control information for playback (e.g., envelope). The break-point function is also fully editable through the GUI. The note heads constructed with ND usually belong to the category (1) or (2). The third option (note head objects) relies on inheritance and this is not yet supported by ND. 3. THE ENP NOTE HEAD DESIGNER There are only a handful of things that need to be defined in order to create a new note head using ND. The required minimum is: Unique name, which in this case is a Lisp keyword (e.g., :tambura, :sprechstimme, etc.), and the note head definition code in Lisp and/or hand drawn graphics composed out of the provided graphical primitives. All additional code and the UI components, i.e, classes, methods, and contextual menu entries, are provided automatically by ND The Components of the Note Head Designer ND is represented by an ND window, which, in turn, contains several specialized components. As can be observed in Figure 4, on the top of the window there is a properties view (1) containing inputs (e.g., note head name and font). Next there is a code view (2) that can be used to enter the note head definition in Lisp. At the bottom of the window there is a graphical tool palette (3), and a preview score/graphics canvas (4). The preview score is fully editable. Additionally, there are also some push buttons to deal with the preview score and the note head definition source code. The ruler above the staff entitled width shows the extent of the note. This can be used as a visual aid when drawing the note head definition by hand. Figure 3. Two bpf note heads. The latter one is in an editable state with the movable break-points shown as black rectangles. Figure 4. The components of the ND window. (1) the properties view, (2) the code view, (3) the graphical tools, and (4) the preview score/graphics canvas. The ruler (labeled width ) showing the extent of the note is displayed above the staff. Also, (5) shows the stem of the chord the designed note is associated with.

130 There are basically two ways to define a note head in ED. First, by using the text-based or code-based approach the user can define the the note heads appearance with the help of the provided set of drawing commands and special variables. Second, drawing by hand using a special graphics canvas and a set of graphical tools Code View In the code view the user can write standard Lisp code (see Figure 4). The users can utilize any of the OpenGL [12] functions provided by LispWorks. Additionally, ENP provides also a set of graphical primitives, e.g., draw-2darrow, draw-2d-box, draw-2d-bracket, draw-2d-circle, draw-2d-line, draw-2d-lines, draw-2d-point, draw-2dpolygon, draw-2d-quads, draw-2d-shape, draw-2d-text, draw-2d-texture, draw-2d-triangles, etc. To be able to use these primitives in a meaningful way, some knowledge about the musical context of the note head is needed. Access to this information is provided through some special variables and objects which are enumerated in the following list: Figure 5. The attributes of ENP note heads: (1) x position, (2) y position, and the widths of two notes of different lengths (3) and (4). of this line are always drawn relative to the position of the note head. The second line of code, on the other hand, draws a line segment from {0.0, 0.0} to a variable end point at {width, 2.0}. The behavior of these two line segments can be observed by comparing the preview scores found in Figures 6 and 7. In the latter score the measure is considerably wider than in the former one, thus giving more room to the note. Notice, how the shorter line segment preserves its length and position but the longer one adjusts itself according to the new width of the note. (1) x and y. These variables give the corresponding position of the note head in the horizontal and vertical axis (see Figure 5). (2) notehead. A character or string defined by the user. (3) width. This variable represents the length or extent of the notehead. As can be seen in Figure 5 this is mainly determined automatically according to the duration of the associated note. It can also be edited by the user through the GUI or algorithmically. Figure 6. Two line segments drawn in the ND canvas. Both have one fixed end point at {0.0, 0.0}. The former one (1) has a fixed end point at {2.0, 2.0} and the latter one (2) has a variable end point at {width, 2.0}. (4) height. Some note heads (e.g., bpf note head) provide information as to their height. For the others this attribute is set to 1.0. This parameter is usually usereditable through the GUI. (5) stem-down-p flag is true if the chord containing the note has the stem downward. (6) pitch gives the midi value of the note. It can also contain a fractional part in case of micro-intervals. (7) augmentation. Optional augmentation (e.g. dots). These are calculated automatically by ENP according to the duration/rhythm of the note. (8) self. This variable gives the actual note object. There are several attributes (in addition to the ones given above) that can be read directly from it. In most cases it is not required to access any information in this way. It is nevertheless provided for advanced applications. Figures 6 and 7 illustrate the use of the width parameter. The two lines of code in the code view both draw a single line segment. The first line segment is drawn from {0.0, 0.0} to a fixed end point at {2.0, 2.0}. The end points Figure 7. The variable end point {width, 2.0} of the longer line segment dynamically follows the width of the note head Graphical Tools In ND there is also a collection of graphical tools that can be used to compose the visual appearance of the note heads. The currently available set of tools consist of the following: (1) Line segments, line strips, arrows, filled or framed polygons, and circles. These tools are quite self explanatory. In general they behave as the ones found in any graphically oriented software. There are, however, some differences. In case of most of the primitives the user can, for example, add and delete points (or vertices) to form more or less complex shapes.

131 (2) Text. This tool allows to add textual information. The user can select the color, transparency, size, and font face. The currently available font faces are Times and Sonata. (3) Line thickness, and stipple can be set to most objects. There is a set of predefined stipple patterns and the user can also enter any arbitrary pattern (a 16-bit pattern determines which fragments of a line will be drawn). (4) Color and transparency. Color and transparency values can be chosen from a palette of some predefined colors and levels of transparencies. Additionally, any color provided by the system color chooser dialog can be used. (5) Textures can be applied to polygons. Currently the texture coordinates automatically map to the vertex coordinates of the polygon (see [12] for further reference on texture coordinates). In the following (see Figures 8 and 9) we give an example how the ND graphics canvas behaves. The note head appearance is always drawn relative to the origo, i.e., the x/y coordinate of the note head. If the note head of the chord containing the note head is transposed or otherwise displaced in the score the graphical objects follow accordingly. This allows to check the behavior of the note head in several different positions. Figure 9. The chord (containing the designed note head) has been manually displaced both in x and y dimensions (transposed up and dragged to the right). The graphical objects adjust their position accordingly. meaning that they retain their shape regardless of their musical context. The third one, in turn, is a dynamic one as it adjusts its visual appearance according to the extent of the note. As the final example we define an algorithmic note head that graphically visualizes the production of a rattling sound A Character-based Note Head This approach is the most simple one. The user can define a character found in either Times or Sonata font faces to be used as the note head. In this example we have selected a c symbol to create a note head that could potentially be used, for example, to represent a pitch-wise constrained note (see [6]). Figure 10 gives the corresponding ND window. Although the definition of the note head itself seems trivial there are nevertheless several things that are handled on the behalf the user. Among other things the drawing method is defined along with the needed menu components. Figure 8. The ND graphics canvas shows some graphical objects with the editable handles shown: (1) polygon, (2) line strip, (3) circle, (4) text, and (5) filled triangle. The origo of the drawing canvas is given in (6). All the graphical objects are drawn relative to this point. 4. SOME NOTE HEAD DESIGNER EXAMPLES In this section we use ND to design several note heads. We begin by defining a simple character-based note head. Then we move to more complex cases as we define three hand drawn note heads. The first two are static ones, Figure 10. A simple character based note head created with ND A Hand-Drawn Note Head Using the set of graphical tools described in section 3.3 makes it possible to draw the note head shapes as they

132 would appear in the score. As our first hand drawn example we define a simple note head composed out of some characters and an arrow (Figure 11). This particular note head means scraping along the stings with the fingernail [10]. The arrow indicates the direction. In addition to the actual appearance of the note head the user needs to define a unique name ( scrape ). to some proportion of it. The Figure 13 shows an example of an object of this kind. The note head could be interpreted to denote, for example, an evolving cluster (e.g., à la Penderecki). Figure 13. A dynamic graphical note head containing some proportionally spaced vertices. Figure 11. A note head with a character component and some simple graphics. Next we use a filled polygon tool to draw a triangle shaped note head (Figure 12). The default polygon object contains four vertices so one vertex is removed simply by selecting and deleting it. One of the remaining vertices can now be dragged to form a triangle shape. Our new note head denotes an inhaling sound as used by, for example, Ligeti in Aventures. This note head is drawn using the filled polygon tool. The polygons, in turn, are constructed out of a set of vertices. A vertex can have two different modes: a normal mode and a proportional mode. Vertices that are in a normal mode have absolute x positions in the coordinate space, i.e., a vertex placed in {1.0, 1.0} remains in that position unless moved explicitly by the user. The proportional vertices, however, calculate their x position according to the space given to the note (width). Note, that the y position of both kinds of vertices is always calculated relative to the y position (read: pitch) of the note, as explained above. Figure 14 gives a closer view of this particular example with the proportional vertices shown. Figure 12. A static graphical note head. As the preview score is fully editable the second note head has been inserted in the score by copying. The latter 1/8 note, in this example, was inserted in the preview score to illustrate the fact that the note heads are ready to be used in the current ENP session. They behave exactly like the built-in ones and can be copied, inserted, and edited even in the preview score. The two previous examples have dealt with relatively static note heads. However, it is sometimes convenient that the note head scales itself according to the space reserved for it in the score. In ENP the note heads can be made to fill the space between two notes or to justify itself Figure 14. A note head drawn using a filled polygon tool. The figure shows also some proportionally spaced vertices An Code Based Note Head As our last example we use the code view to define a Lispbased note head. We implement a note head for a rattling effect as described for example in [2]. The code is used to join a series of Bézier curves to form a complex, evolving curve that completely fills the space reserved for the note head. Let us examine the source code in more detail (see code view in Figure 15). First, we define the outer contour of the shape and define a scaling factor for each of

133 the Bézier curves (y-shape, y-scaling). We basically draw the same curve repeatedly in a loop each time with a different transformation and scaling values (see the two OpenGL macros with-gl-translate and with-gl-scale in the bottom of the code view). Every other curve is also flipped vertically, using a negative scaling value, to smoothly join the end of each curve with the beginning of the next one. The whole shape is scaled inside the extent of the note by dividing the width of the note by the number of individual curves in the note head shape (the length of the y-shape). The preview score in Figure 15 shows the final note head shape. build-in ones. Finally, there are some improvements planned in ND: (1) Usually the note heads may have slightly different parameters and offsets depending whether they are printed stems up or stems down. The ND window could potentially be organized so that there are two different detail views (one for up stem and another for down stem) and one overview which would display both versions at once. Currently it is not possible to distinguish between up or down stem variants other than using the code based approach. (2) There should also be a possibility to group the note head entries in the menus, much in the same way as expressions are currently grouped [4]. This would make it more convenient to write dedicated libraries for specialized cases such as modern notation or shape-notation. (3) There are still some useful tools missing. (a) Grouping/Un-grouping. With this tool the user could group several objects to form a unit that could then be moved simultaneously. (b) Rotation. All objects should be allowed to be rotated in a 2D space either interactively or additionally by providing a rotation angle. (4) There should be a collection of more dedicated graphical tools such as triangles, squares, etc. This would allow more specialized behavior, thus, when zoomed, a square would retain its shape and so on. Now all the points forming a polygon can be moved individually. Figure 15. A complex note head shape composed out of several Bézier curves. 5. CONCLUSIONS A new visual tool called ENP Note Head Designer was introduced along with several workable examples. The general note head scheme of ENP was also covered. The visual interface described in this paper allows to define note heads using either a text-based or graphically oriented approach. This allows users with different backgrounds in programming to take advantage of this tool. Furthermore, the uniform representation of note heads in ENP allows several advantages. First, it is possible to use the new note heads in the current ENP session without any compiling or reloading phase. Second, it is also possible to use the new note heads along with ENP-score-notation, the text based format of ENP (see [5, 7]). There is no need for any additional coding; the note heads behave as any of the (5) A grid with snap-to-grid option should probably be offered. There should be the possibility to choose individual vertical and horizontal grid spacings (e.g., proportions of the width horizontally, and fractions of the line spacing or staff height vertically). (6) Finally,there should also be a possibility to save the current ND session. This would allow to recall a note head in an editable form and continue to work with it. Now, only the note head definitions can be saved in a file. 6. ACKNOWLEDGMENTS The work of Mikael Laurson has been supported by the Academy of Finland (SA ). 7. REFERENCES [1] MakeMusic! Inc. Finale User Manual. [2] Erhard Karkoschka. Das Schriftbild der Neuen Musik. Hermann Moeck Verlag, Celle, 1966.