Constellation: A Tool for Creative Dialog Between Audience and Composer Akito van Troyer MIT Media Lab akito@media.mit.edu Abstract. Constellation is an online environment for music score making designed to intuitively explore and asynchronously co-create a music piece by drawing constellations of lines that change the arrangement of musical materials composed by Tod Machover for A Toronto Symphony. This web environment, comprised of hundreds of colorful graphical dots that represent fragments of a single piece of recorded music, explores a number of challenges in designing musical participatory experience. The design empowers participants to be musically expressive by encouraging them to communicate, co-create, and co-explore. The interface also addresses a unique challenge in making the scoring process intuitive by the use of real-time explorative concatenative synthesis so that the participants do not need to learn about or be skilled at music-scoring. The musical scores created by the website visitors were eventually used by Tod Machover to compose the final version of A Toronto Symphony. Keywords: Music, Composition, Compositional dialog, Collaboration, Symphony, HTML5, Co-creation, Concatenative Synthesis 1 Constellation in the context of A Toronto Symphony A Toronto Symphony is a symphonic work commissioned by the Toronto Symphony Orchestra that premiered in March 2013. The work explored a new musical ecology to create a true creative and collaborative environment between experts and amateurs in the making of the final piece of music. Furthermore, we invited the entire city of Toronto to create a new symphony. The final piece of music was composed through a series of infrastructures built for people in Toronto to participate in, including a variety of web-based music composition applications, a social media framework, and on-site community-building activities (e.g. workshops, school activities, and city explorations) to bring together an unprecedented number of people from diverse backgrounds. This process established a new model for creating complex collaborations between experts and non-experts in composing a piece of music. A Toronto Symphony envisioned that a much more fluent communications between people listening to music and people making music could happen in this way. Constellation, along with other web applications, was created as a part of A Toronto Symphony to invite anybody who wanted to collaborate and participate in creating a piece of music from scratch.
2 Akito van Troyer Constellation aimed to creatively engage audience members to become part of the compositional process. This web environment then helped incorporate their creative activities into the final live concert performance for A Toronto Symphony. Website visitors, who did not need to have specialized musical training, used a visual interface that consisted of hundreds of colorful graphical dots that represent fragments of a single piece of recorded music to choose which musical materials to work with and to arrange and modify the musical materials by clicking and dragging on top of the interactive visual score (see Fig. 1 top). The visitors were asked to create constellations of lines connecting sounds which in turn became their musical score (see Fig. 1 bottom left). Musical scores created by the visitors could be shared among other people through a score gallery so that visitors could listen to their composition and to remix each others work. Fig. 1. Top: The interface of Constellation that visitors see first when they access the website. Bottom left: The same interface with constellations of lines. Bottom right: The same interface with a different dot orientation. 2 Goals and Challenges In designing Constellation, we explored how an online music score-making environment can facilitate dialog between an audience and a composer to the point that they could both contribute to creating a final piece of music. For this musical dialog to happen, we envisioned that the score-making environment ought to encourage participants to be expressive and encourage them to communicate, co-create, and co-explore. Similar visions can be observed in many of the
Constellation: A Tool for Creative Dialog Between Audience and Composer 3 prior works similar to Constellation. These include interactive musical systems where website visitors composed music through open-form music scoring systems which then were used for performances (Piano Etudes and Graph Theory) [4]; where participants made sounds that were then mixed and processed in real-time over broadcasting system [11][6]; and where links between online environments and live concert performances in real time remotely influenced the performance (Brain Opera and Cathedral) [8][2]. These music systems attempted to enhance the musical experience of participants and gave new perspectives to the prospect of dialog between an audience and a composer or between amateurs and experts. To enable anyone to score a piece of music, the interface for the scoring process must be intuitive enough so that the participants do not need to learn about or be skilled at music-scoring. One way of addressing this challenge is to use a sketching or drawing metaphor to make the composition interface and process simpler for novices. For example, the Hyperscore graphical computerassisted composition system enables novices, particularly children, to intuitively compose western traditional music through freehand drawing [3]. In this scoring system the act of sketching literally becomes novices musical piece. Other graphical composition software systems that promoted similar drawing method as a way to score a piece of music include UPIC [20], Different Strokes [21], and CyberBand [19]. Constellation also uses the act of drawing as the metaphor of composing. Participants draw constellations of lines on top of the graphical environment to compose a piece of music in which the synthesis method is based on a real-time explorative concatenative synthesis. An explorative concatenative synthesis was used as the base synthesis engine of Constellation for music making. In addition, from a technical perspective, Constellation also explored implementing a concatenative synthesis engine on web browsers that support Web Audio API [18] proposed by W3C or Audio Data API by Mozilla [10]. Concatenative synthesis is a technique for synthesizing sounds using segments of audio data to assemble and construct the resulting sound [13]. Furthermore, explorative concatenative synthesis refers to a method of synthesis based on specifying a target as a multidimensional curve in the descriptor space [13]. Concatenative synthesis techniques for music have been explored since the 2000 in particular through the work of Schwarz and Pachet [15][22]. Different approaches to concatenative synthesis have also been explored by many previous works [9][7]. More recently, the creative, expressive, and explorative use of a concatenative synthesis engine for performance and composition has been explored by projects such as the Sandboxes [16], AudioGraden [5], and Eargram [1]. All of these projects demonstrate that applying a concatenative synthesis technique to expressive composition and performance is possible and can result in unique music. Like many of these projects, Constellation linked the compositional process with concatenative synthesis. To our knowledge, Constellation is the first web browser-based application that used real-time concatenative synthesis as a running synthesis engine for an asynchronous collaborative and creative music composition.
4 Akito van Troyer 3 Design and Implementation The interface of Constellation is designed in a way that engages users to explore a soundscape with mouse gestures before they draw their musical scores. After exploring the environment, users can draw constellations of lines to create their own composition. This core flow of interaction in Constellation is illustrated in Fig. 2. We can interpret this as a representation of the universal Model-View- Controller (MVC) design pattern often used in computer science for building a computer software program. The Model is the concatenative synthesis database which consists of audio features associated with each segment of an audio file. Both the synthesis engine and visual interface relies on the database to synthesize sound and draw graphical dots. We can consider them to be the View of Constellation since these are what users see and hear. The control input is mediated through the user s mouse gesture on the graphical interface. From the mouse gesture, we acquire the two dimensional mouse coordinates at all time when the mouse is being pressed down. We use this information to calculate the mouse acceleration to determine how many dots are to be selected at any given time. The k-d tree map looks for the closest graphical dots to the current mouse coordinates and passes their identities to the synthesis engine. The synthesis engine then generates and sequences the correct sound fragments based on these identities. Fig. 2. The core interaction flow of Constellation. 3.1 Audio Analysis and Synthesis Constellation relies heavily on the data collected from the analysis of a source sound file to synthesize sound. We first blindly segment the sound file into subfiles at an arbitrary length, then extract audio features for each segment based on standard audio signal processing methods and audio feature extraction methods described in [12]. 14 features related to spectrum, pitch, and temporal information are extracted in this process in addition to the first thirteen coefficients of
Constellation: A Tool for Creative Dialog Between Audience and Composer 5 Mel-frequency cepstral coefficients (MFCCs) for each segmented sound. The extracted features are then normalized and stored in JavaScript Object Notation file format (JSON) to asynchronously load the data on the web browser. The web application uses the analysis data to draw graphical dots on the interface and to also synthesize sound when the user interacts with the interface. Constellation uses an explorative audio concatenative synthesis technique to rearrange precomposed musical materials and create a digital music composition. It is important to note that our synthesis process does not involve a musical target model unlike many concatenative synthesis techniques described in the Section 2. Instead, the target sound to generate is determined by the mouse position and gesture of the user, and the sound fragments or grains are synthesized using a similar technique to the asynchronous granular synthesis process. This browsing approach is very similar to the Caterpillar system proposed by Schwarz [15]. We implemented such a synthesis engine in Javascript using Audiolet [17], a graph based audio routing API that encapsulates Web Audio API and Audio Data API. Audiolet has prebuilt sound generators and effects, and a sample accurate scheduler can be used to precisely control the timing of synthesis. Audiolet is also an easily extendable language, so we built custom sound generators to realize concatenative synthesis in the web browser environment. 3.2 Mapping The position, size, and color (red, green, and blue) of graphical dots are computed by using the feature set extracted through the audio analysis process on a 2dplot interface similar to the interface of EarGram [1], [5], and [14]. Six features are arbitrarily selected out of 27 features. Although users are not informed about which feature combinations they are using, they have the choice how to arrange the graphical dots by toggling pattern buttons shown in the left side of Fig. 3. The unit selection algorithm for the synthesis engine is the k-d tree algorithm that looks for the distance between a mouse coordinate and coordinates of graphical dots. Units are selected from the database that best match the given target unit descriptors according to the Euclidean distance. The acceleration of a mouse gesture is used to determine how many dots at a given time is selected for playback. Slower mouse movement results in selecting fewer dots so that the user can focus on exploring individual sounds. The closest dots to a mouse pointer visually becomes larger to indicate to the users which dots they have currently selected for playback. Fig. 3. Left: Five pattern buttons that reconfigure the orientation of graphical dots. Right: Buttons needed for scoring and uploading a score.
6 Akito van Troyer 3.3 Composition Process When the user is done exploring and decides to steer a path through the environment as a score, the coordinates of each point where a line segment anchors are stored as JavaScript variables. The users can then edit individual line segments or delete the entire line. All scoring modes are toggled using buttons located at the top right corner of the interface (see Fig. 3, right). Once lines are in the desired formation, the user can play back their composition using the playback control panel located at the bottom of the interface. When the users are satisfied with their composition, they can upload their score by using the upload button also located at the top right corner (see Fig. 3, right). After the score is submitted, users can share their scores and opinions through social media platforms such as Facebook, Twitter, and Disqus. 4 Discussion The compositions made by participants served as a useful way to evaluate our design and implementation decisions. Furthermore, we had several informal discussions with novices and experts who interacted with Constellation that led to some interesting discoveries. Novices were more inclined to play and explore the environment with their mouse gestures to understand the musical world provided to them. They often mentioned that Constellation encouraged them to actively discover new sound collage patterns, leading to motivating experiences throughout the session, but they were less oriented towards drawing lines to make their composition. On the other hand, expert musicians often found themselves trying to understand the mechanism of the entire system. After taking their time to explore the environment, they were eager to create their music score in the way they specifically wanted. They were able to find sound textures that they liked and to remembered the pathway that would produce the satisfying result that they aimed for. Some experts complained that they wanted to have more detailed control of the synthesis engine such as pitch shifting, grain duration, and grain generation behavior. Other experts wanted to have more control over the visual aspects such as the ability to zoom into a particular area of graphical dots to listen to individual sounds more carefully. The majority of both novices and experts who interacted with the interface found that Constellation to be captivating and expressive, and they indicated that navigating the interface was comfortable and intuitive. Most of them specifically mentioned that the use of simple and expressive mouse gestures to produce rich and dynamic musical results was the key factor in making the interaction motivating and enjoyable experience. They also mentioned that this feature is what lead them to quickly grasp the interface and smoothly navigate through the compositional environment. While the experience of manipulating sound in the environment was good, some novices have expressed to us that they did not submit their scores and share them with other people. Their reasons for not submitting their scores were because they felt that their composition did not meet
Constellation: A Tool for Creative Dialog Between Audience and Composer 7 the quality of what the composer was looking for or because they did not even realize that they could submit their composition to share with other people. For providing a communicative, co-creative, and co-explorative environment, both novices and experts agree that Constellation achieves this to a certain degree. For instance, some novices mentioned that they could quickly learn how to use Constellation through interacting with other people s compositions submitted to the score gallery. Some even took this further and remixed existing scores by adding and deleting lines before resubmission. This confirmed us that Constellation can facilitate co-creation and co-exploration. However, making the environment communicative and facilitating conversation among participants appeared to be a harder problem. Some participants mentioned that they did not want to use the social media platforms integrated with Constellation to share their scores and preferred to keep the conversation within the environment and the score gallery. Others mentioned that they would prefer to share their music score if the communication platform was either in a blog or a chat client format. 5 Future Work We acknowledge that Constellation requires formal user studies to quantitatively evaluate how explorative, expressive, and intuitive the interface is. As we conduct the user studies, we would also like to test whether providing more seamless, flexible, and reconfigurable interface can help novices in sharing their creation with others and making increasingly complex compositions. For example, we think that making the uploading process more intuitive and uninterrupted (e.g. saving the composition as users draw constellations) could potentially accelerate a better sharing experience among novices. Furthermore, we think that providing novices an ability to rearrange and tailor its interface, visual representation, and sound synthesis engine could help novices in making much more serious compositions. For instance, currently participants only have a choice to rearrange musical materials provided by us; we would like to enable them to be able to upload their own favorite music to better motivate them to use Constellation. We also would like to give participants better access to the underlining database to create opportunities to reconfigure graphical dots in whatever way they wish to make their visual look and music score sound more unique. Finally, we would like to give participants more access to the synthesis parameters through their use of mouse gestures. 6 Acknowledgement Constellation is available at http://toronto.media.mit.edu/scores/. The author would like to thank Muriel R. Cooper Professor Tod Machover, Simone Ovsey, Peter Torpey, and Ben Bloomberg for their help in the creation of Constellation. Thanks to the members of the Opera of the Future group at MIT Media Lab for your moral and editing support.
8 Akito van Troyer References 1. Bernardes, G., Guedes, C. and Pennycook, B.: Eargram: an application for interactive exploration of large databases of audio snippets for creative purposes. In: Proceedings of the 9th International Symposium on Computer Music Modelling and Retrieval (2012) 265 277 2. Duckworth, W.: Making music on the web. Leonardo Music Journal (1999) 13 17 3. M.M. Farbood, E. Pasztor, and K. Jennings.: Hyperscore: a graphical sketchpad for novice composers. IEEE Computer Graphics and Applications 24 (2004) 50 54 4. Freeman, J.: Web-based collaboration, live musical performance and open-form scores. International Journal of Performance Arts and Digital Media 6 (2010) 149 170 5. Frisson, C., Picard, C. and Tardieu, D.: Audiogarden: towards a usable tool for composite audio creation. QPSR of the numediart research program (2010) 33 36 6. Jordà, S.: Faust music on line: An approach to real-time collective composition on the internet. Leonardo Music Journal (1999) 5 12 7. Kersten, S., Maestre, Esteban. and Ramirez, Rafael.: Concatenative synthesis of expressive saxophone performance. In: Music Computing Conference (2008) 8. Machover, T.: Brain opera http://brainop.media.mit.edu (1996) 9. Maestre, E., Ramírez, R., Kersten, S. and Serra, X.: Expressive concatenative synthesis by reusing samples from real performance recordings. Computer Music Journal 33 (2009) 23 42 10. Mozilla.: Audio Data API. https://wiki.mozilla.org/audio_data_api (2013) 11. Neuhaus, M.: The broadcast works and audium. http://auracle.org/docs/ Neuhaus_Networks.pdf (1994) 12. Peeters, G., Giordano, B. L., Susini, P., Misdariis, N. and McAdams, S.: The timbre toolbox: Extracting audio descriptors from musical signals. The Journal of the Acoustical Society of America 130 (2011) 2902 2916 13. Schwarz, D.: Corpus-based concatenative synthesis. IEEE Signal Processing Magazine 24 (2007) 92 104 14. Schwarz, D., Beller, G., Verbrugghe, B. and Britton, S.: Real-time corpus-based concatenative synthesis with catart. In: Proceedings of the COST-G6 Conference on Digital Audio Effects Montreal, Canada (2006) 279 282 15. Schwarz, D.: A system for data-driven concatenative sound synthesis. In: Digital Audio Effects (2000) 97 102 16. Tremblay, P. A. and Schwarz, D.: Surfing the waves: Live audio mosaicing of an electric bass performance as a corpus browsing interface. In: Proceedings of the 2010 International Conference on New Interfaces for Musical Expression (2010) 17. Audiolet.: https://github.com/oampo/audiolet (2013) 18. W3C.: Web Audio API.: https://dvcs.w3.org/hg/audio/raw-file/tip/ webaudio/specification.html (2013) 19. Wright, J., Oppenheim, D., Jameson, D., Pazel, D. and Fuhrer, R. M.: Cyberband: A hands on music composition program. In: Proceedings of the International Computer Music Conference (1997) 383 386 20. Xenakis, I.: Formalized music: thought and mathematics in composition. No. 6. Pendragon Press (1992) 21. Zadel, M. and Scavone, G. Different Strokes: a prototype software system for laptop performance and improvisation. In: Proceedings of the 2006 International Conference on New Interfaces for Musical Expression Paris, France (2006) 22. Zils, A. and Pachet, F. Musical mosaicing. In: Digital Audio Effects (2001)