Musictetris: a Collaborative Composing Learning Environment Wu-Hsi Li Thesis proposal draft for the degree of Master of Science in Media Arts and Sciences at the Massachusetts Institute of Technology Fall 2007 Thesis Advisor: Barry Vercoe Professor of Media Arts and Sciences MIT Media Laboratory Thesis Reader: Tod Machover Professor of Music and Media MIT Media Laboratory Thesis Reader: Mitchel Resnick LEGO Papert Professor of Media Arts and Sciences MIT Media Laboratory Thesis Reader:
Motivation Composers seem to compose alone. It is not just a stereotype especially in western music culture, in which composing is mostly an off-stage process. A composition is published or performed only after the composer finishes endless revision. However, in different music cultures, composing could mean to improvise on-stage, which could happen between multiple improvisers. As various graphical computer-assisted composing tools gradually become pervasive, composing is no longer exclusive to professional composers [1][2]. A missing element of those tools is the chance to share and collaborate through the composing process. A beginner in composing will enjoy composing even more if not composing alone. Why don t they learn to compose together, in an improvisational style? Why can t they share a tiny catchy tune that just comes out their minds? In this thesis, we seek to fill in the missing piece in the proposed composing environment. Aims and Objectives The focus of the thesis is to explore how the design of a composing environment can encourage sharing and collaboration between learners through the composing process and how such collaboration would affect the quality of learning. Sharing can happen during the entire composition, even if it s just a short tune. The challenge is to design a composing environment that makes users intuitively want to share and reuse music fragments. To make this happen, we shape the music fragments into playful bricks, so that they can piece together the composition by playing with the bricks. We propose Musictetris, a graphical composing learning environment for novice composers of all ages. It is more of a peer-assisted composing environment than a computer-assisted composing tool. A user study will be conducted to evaluate how Musictetris reshapes the learning process of composing. The contribution of the thesis is to discover the roles of sharing and collaboration throughout the composing process in a computer-assisted learning environment. Musictetris will be the first realization following such concept.
Related Work Among various graphical computer-assisted composing tools, Hyperscore is arguably the most complete project, which takes a lot of musical perspective into the design [1]. It provides a layer of abstraction between two levels: the melodic, note-level input as motives and large-scale, form-level shaping. The two-layer graphical control and the tension line allow users to conceive composing ideas through a top-down manner. Their website enables the Hyperscore community to share their compositions with others. Besides, Hyperscore also realized collaborative mode for classroom situation. Musical Sketch Pads uses a straightforward approach in graphical interface design, mapping the pixels into musical notes according to its position [2]. The interface is inherently a pixel map graphical editor, which allows users to compose in a bottom-up fashion. Users not only can draw in free-hand style, but also edit the score by moving, reversing, and copying pixels in a selected region. While our design follows the straightforward bottom-up composing interface design, we want the users to compose with a structural sense. The Musictetris guides the users to a two-stage composing: (1) compose the motives (basic composing pattern), and (2) repeat and develop the motives. The design attempts to capture Hyperscore s top-down view while remain clear and simple in the interface. The other important element in the thesis is to open up the chance of sharing and collaboration through the composing process. Scratch is a programming learning environment which accomplishes the goal [3]. Other than fulfilling sharing and collaboration in composition domain, Musictetris enable users to share and reuse in music fragment level, rather than the entire composition. It will be interesting to observe how such feature reshape the interaction between users. Approach 1. Graphical Interface Design The essential part of a graphical composing interface design is the mapping mechanism between the graphical and musical features. The general issues include: How does the visualization reflect the music in different levels? Is the mapping easy for users to understand? Is the input interface easy to play with? Does it open to precise control?
Musictetris adopts a one-layer direct mapping between canvas and score, where the horizontal and vertical axes represent time and pitch, respectively. The basic graphical unit on the canvas is a block, which is interpreted as a musical note according only to its position and color. A block on the canvas is either an individual note, or part of a note when it is horizontally connected to other blocks. Each color is assigned to a music instrument or specific player. It is intrinsically the piano roll representation except that we apply nonlinear scale to the mapping of both axes. Such nonlinearity can be applied to embed musical scale and rhythm in the graphical composition. The graphical interface displays every note as a colored bar, which makes it easily understandable to users, and it is also technically feasible to revise every single note on the canvas. The drawbacks include: (1) the interface is not capable of abstracting music in a higher level, e.g., harmony, and (2) it s difficult for users to control precisely with freehand drawing. Therefore, we create different embedded musical scale in the pitch axis so that it is much easier for users to draw musically consistent melody in freehand. In addition, our embedded musical scale also turns the relatively scattered grids in piano roll representation into more connected visual patterns. It would be easier for Fig. 1: The graphical interface with a typical users to group those patterns and observe freehand drawing-style composition. Each color musical structure from the visualization. represents a music instrument. 2. Music bricks Freehand drawing is a great first lesson for novice composer, but a composition created only by freehand drawing is not far from random music since the interface does not interpret the drawing in higher level. Therefore, the second lesson here is to create patterns as composing material, which we name as music bricks. The music bricks are the motives and material for composing, representing basic patterns of melody, chord, or rhythm, as in Fig. 2. Each brick consists of single or multiple
blocks, which don t necessarily need to be physically connected. In addition, multiple bricks can be merged into a compound music brick in an overlapping or interweaving manner. Fig. 2: From left to right, a melody, rhythm, and compound brick. Note that although these bricks are not physically connected blocks, they are still considered as virtual bricks. The naming of music bricks and the visually brick-like music pattern deliver important metaphorical messages to the users. First of all, they tend to play with those music bricks in a way similar to what they do with Lego bricks: moving, combining, rotating and so forth. Since they can hear the sound of each music brick, they learn the musical meanings behind every operation on the brick. The most musically fancy operation is to rotate a music brick, which exchanges the pitch and rhythmic properties of a music brick. Rotating a chord brick would produce specific rhythm and vice versa. The music brick is the basic unit for sharing. Users don t need to create all the bricks before composing. They can collect existing bricks from somewhere. However, if they do create novel music bricks, they can share them to others. Users can create containers to collect bricks of different category: the melody box, the rhythm set, the Simpson s or the Super Mario theme collection and so on. Our design makes it intuitive for users to share and reuse music fragment. The brick-playing nature also induces users to compose together. Each of the users can hold their favorite bricks, trying to piece together the composition collaboratively. Furthermore, they can improvise music by jamming with other players. These bricks can be considered as hyper-notes. In this way, the basic unit of performing is not necessarily a note, but one of these music bricks. The improvisation can be recorded as an inspiration for composing. 3. Discussion (1). Playing with different musical scales User can select one out of six different embedded musical scales in vertical axis of the canvas. These scales include major, minor, chromatic, whole-tone, pentatonic and blue note scale. It is especially intriguing to only change the mapping without reposition
the grids, which seems to shape the music into different culture in a simple way. (2). Music trees In Musictetris, compositions can be easily co-composed or contributed by several authors because the design encourages users to re-use music bricks in different pieces. It would be interesting to analyze the re-use history of those bricks. For example, a combination of two bricks becomes a new brick, and the new brick is used in a composition. We can use a tree or graph visualization to track the relation between bricks, compositions and composers. It would really be encouraging if a composer would know his/her little music brick appears in many different pieces by others. Evaluation The complete thesis will include the implementation of Musictetris which runs on PC and OLPC. A user study will be conducted on children in elementary school. We will try to observe how Musictetris reshape the learning process of composing, how effective Musictetris is in triggering children s interest in composing and how its social feature contribute to their interest. Schedule of Work Nov & Dec 07 & Jan 08 Design, Implementation, Testing Jan 08 Submit experiment protocol to COUHES for review Feb 08 Recruit subjects and run the experiment Mar 08 Analyze experiment data and begin writing April & May 08 Complete writing of thesis. Incorporate reader comments. Bibliography 1. Farbood, M., Kaufman, H., Jennings, K. (2007). Composing with Hyperscore: an Intuitive Interface for Visualizing Musical Structure, Proceedings of the ICMC. 2. Subotnik, M., Musical Sketch Pads online activity. http://www.creatingmusic.com/mmm, 1999-2007. 3. Resnick, M. (2007). All I Really Need to Know (About Creative Thinking) I Learned (By Studying How Children Learn) in Kindergarten. Proceedings of the ACM SIGCHI conference on Creativity & Cognition, Washington, DC.