The Musicat Ptaupen: An Immersive Digitat Musicat Instrument Gil Weinberg MIT Media Lab, Cambridge, MA, USA Abstract= A digital musical instrument, the "Musical Playpen", was developed in an effort to provide children with immersive musical experiences. The instrument senses children's motion inside a playpen filled with plastic balls and maps their gestures to musical output. This allows the players to use familiar and fun play activities while exploring the musical consequences of their actions. Observations of children's play patterns in the instrument suggest that immersive musical experiences can introduce infants and toddlers to a variety of creative and expressive musical experiences, rarely offered by traditional musical instruments. Keywords: Children1; Expressiveness; Immersion; Musical instrument 132 i. Theoretical Background Canonical cognitive development theories affiliate constructive reasoning with infants' early sensorymotor stage [1]. According to these views, the concrete nature of constructive thinking enables children to build their knowledge by accommodation and assimilation, through interacting with physical and cognitive "building blocks." Many traditional musical instruments present players with such constructive musical "building blocks" by providing an infrastructure for an expressive assembly of pitches, volume and duration values into melodies. Various musical education methods also use constructive methods in an effort to provide young children with fun musical activities. For example, the Orff Schulwerk method [2] encourages children to construct musical compositions from enjoyable and familiar activities like singing, clapping, and tapping on objects. The Dalcroze Eurhythmics method [3] puts an emphasis on constructing kinesthetic bodily gestures into dance and musical pieces. These bottom-up, constructive musical experiences can be valuable for musical learning as well as for composing, performing and listening. Constructive musical activities (and instruments), however, are rarely designed to elevate the player towards an immersive musical experience, which is based on the notion that the musical whole cannot always be perceived as the sum of its components. Such activities can also rarely be enjoyed by infants and toddlers due to the level of logical reasoning and physical skills that they require. The Musical Playpen is designed to address some of the deficiencies that are presented by the constructive approach. The instrument is based on an "immersive," top-down method, which places the child in a fully holistic musical environment that can be deconstructed and explored. The concept of Musical Immersion is informed by Susan Langer's definition of "Presentational" and "Discursive" symbolic systems [4]. According to Langer, artistic experiences are better perceived by a Presentational symbolic system, which focuses on processing simultaneous, rich emotional stimuli. The discursive symbolic system, on the other hand, is more constructive and analytical in nature. Eanger claims that an artistic experience cannot be represented by a discursive linguistic system since "An idea that contains too many minute yet closely related parts, too many relations within relations, cannot be 'projected' in discursive forms; it is too subtle for speech" [4, p 93]. This leads to the affiliation of high-level presentational symbols with a holistic experience such as Csikszentmihalyi's "Flow" [5], an experience that is devoid of discursive representation and analytical reasoning. The utilisation of fun and familiar play activities is an important feature in providing such immersive musical experiences for infants and toddlers. At these young ages, children can experience difficulties in creating an immediate relationship with new objects or environraents. This led me to formalise a system in which musical enhancements are implemented in familiar and already loved activities. Such implementations can allow children, who are affectively immersed in the musical 9 Springer-Verlag London Ltd Personal Technologies (1999) 3:132-136
experience, to gradually discover the musical effect of their motions and navigate their gestures into musical domains. It also can allow children to enjoy an expressive musical experience without undergoing a long knowledge and skills construction process. It is relatively easy to conceptualise a static immersive musical environment, where children are provided with high-level abstract causality between their gestures and an ambient audible output. A much more difficult challenge is for the environment to encourage children to explore the constructive musical infrastructure that is in the system's core. Players should be able to use the amorphous nature of the immersive experience as a starting point for a top-down musical exploration that can also bear an educational value. Hence, the Musical Playpen's application is designed to allow children as young as one year old to create musical pieces using familiar "presentational" play gestures as well as to deconstruct their musical environment and gradually internalise basic musical concepts such as pitch, tempo timbre, phrasing, etc. 2. Software and Hardware Implementation The Musical Playpen is a whole-body enlargement of the Musical Candy Bowl, a palm size immersive prototype 1. The playpen enhances this first prototype by providing embracing physical immersion in addition to the cognitive immersion provided by the software. In order to allow for gradual musical internalisation of fun play gestures, it was decided to use a 5 x 5 ft playpen filled with 400 colourful plastic bails, as a space that is compelling for children even without musical association. Players' movements around the playpen propagate from ball to ball and trigger four piezo-electric accelerometers hidden inside four selected balls in each corner of the playpen. The balls' ability to transmit hits to their neighbours as well as the accelerometers' high sensitivity allow for almost any delicate movement around the playpen to be captured by at least one sensor. The analog signal is then digitised by an I-cube digitiser 2 and sent to a Macintosh computer running Max 3 - a graphical 1Developed by the author and Seum-Lim Gan. More information on the Musical Candy Bowl can be found at http://www.media.mit.edu/-gili/research/immersive/ immersive.html#candy 2The I-cube web site - http://www.infusionsystems.com/ 3Max web site - http://www.opcode.com/products/max/ programming language especially designed to receive, interpret and map control data to Midi [6] devices. The Max program parses and analyses the digitised data, based on the players' position, location and level of energy around the playpen. The program then uses graphical objects that control logical operations, data handling, timing, arithmetic and system devices operation to send Midi instructions to a Roland Sound Canvas 4 synthesiser, which generates sequences of melodic and percussive sounds based on the application's algorithm. The application,naps two of the playpen's corners to a pitch look up table that represents an Indian Raga scale. The more accelerated the body movements in these comers, the higher the Indian raga pitches get. The other two comers control a percussive section where sensor acceleration is mapped to a drum sound lookup table. The drum table contains low drums in the lower range (bass drums, tablas), higher frequency percussive sounds in the middle range (snares, tam-tams) and high frequency sounds in the highest range (high hats, cymbals, crashes). For both look up tables, a low-range random generator provides velocity and duration values in order to generate a more dynamic musical output. Since it is difficult to control specific sensors individually (unless being told, the player does not know in which balls the sensors are hidden), an "immersive" effect of generating holistic musical output in relation to interdependent movements is created. However, due to the simplicity of the basic algorithm (one-to-one mappings between the level of acceleration and the pitches' height), interested children can gradually explore the exact connections between their gestures and the musical output. By learning to control their play gestures in the Indian raga comers, children can internalise musical concepts such as scale, pitch and contour while expressively experimenting with creating their own "Indian phrases." By accurately manipulating the drum corners, children can experiment with different drum sounds and internalise concepts like frequency and timbre, while exploring the causality between fun play gestures and rhythmical patterns. 3. Observations and Reflection The playpen was designed to encourage children to participate in top-down immersive musical 4Rolland web site - http://www.rolandus.com 133 The Musical Playpen: An hnmersive Digital Musical Instrument
- p [] ~ ~ The Mus,cal Playpen Max arch ~ - - E g _ - - :? - ~i!'7 - r i 134 m eoot i Im ken~ i s~l Moni'tor I 4, [ ~ 5:, Fig. 1. The Musical Playpen max patch. Acceleration values from the four corners are received, parsed and analysed by the CubeBox4al04 object. Based on the sensor data, the object then sends Midi note-on commands to two raelodic and percussive lookup tables. The ppvelocity and ppduration objects provide random values to the makenote generator in order to create a more dynamic output. experiences. Its immediate responsiveness and its tendency to arouse tempestuous play gestures suggested that children would start their experience in an immersive holistic way and gradually expand towards lower level exploration. However, the observations that were conducted at MIT and at the Boston Children's Museum from 1998 to 19995 have shown greater diversity in the response of children to the new instrument. For example, one i-yea>old infant started her session with a 5A video clip of the observations can be found at http:// www.media.mit.edu/- gili/playpenweb.mov careful investigation of the different corners' functionality. The first sequence of notes that she "played" was generated when she was placed near an Indian-Raga corner. The infant looked at the direction of the sound source (the speakers were hidden under the balls in the playpen) and tried to move her hand in that direction, seemingly trying to repeat the sound she heard. When she succeeded and another sequence was played, she smiled, took one ball and tried to shake it in her hand, obviously without success. Frustrated, she then threw the ball back towards another comer, generating a different percussive sequence. She G. Weinberg
approached the new corner while moving her torso back and forth, laughing loudly when finding out that her movements controlled the music. After stopping for a while, as if she were considering her next move, the infant started to move her body again, very slowly, back and forth. Gradually her movements became faster, generating higher and higher percussive frequencies. She then stopped abruptly and waited, maybe processing the new connections that she found. Only after repeating this behaviour in another corner, did the infant seem to be ready to use more expressive, less controlled gestures all over the playpen. She now seemed to have fun. This bottom-up, almost analytical, constructive approach did not repeat itself with another 16- month-old toddler. The first play patterns that were demonstrated by him were turbulent movements all over the playpen, kicking and waving his arms, throwing balls all over and accompanying himself by singing and screaming joyfully. The toddler was immersed and excited by this flow-oriented activity and did not allow anyone to stop him or take him out. After this expressive explosion, the toddler gradually started to explore the different responses in the different corners around the playpen. He then performed several abrupt jumps from one corner to another. Towards the end of the observation, the toddler seemed to have developed unique play patterns and a personal musical tendency: His "compositions" included ecstatic random parts in the centre of the playpen which were interrupted by gentle exploratory parts near the different comers. In a controlled environment, where he was placed in a playpen that was disconnected from its musical output, no organised play patterns were observed. The above observations exemplify the diversity in children's responses to the playpen. Some Fig. 2. A toddler plays in the Musical Playpen. players preferred staying in the immersive stage and its expressive, holistic, sometime ecstatic nature. Others concentrated on an analytical exploration process of the low-level musical parameters that construct a full musical experience. In several observations, the playpen managed to be successful in encouraging children to develop their play patterns over time. A number of "bottom-uppers" were able to elevate themselves towards new immersive realms. A number of"top-downers" also managed to "dive" down and explore lower-level musical aspects. The above findings suggest that players' tendencies and personal play patterns serve an important role in defining the immersive (or constructive) nature of the musical experience. Even in an instrument like the playpen, which was designed to be played in a top-down immersive manner, a "constructivist" player will be able to find the low-level musical building blocks first, and only then construct them into a full musical experience. 4. Conclusion and Future Work t have described how an immersive musical instrument can provide infants and toddlers with a variety of musical experiences by combining expressive and educational applications in fun and playful activities. I hope that immersive instruments, such as the Musical Playpen, will be able to fill a void in the musical experiences currently being offered to young children. I intend to design an experiment that will test the instrument's educational value. I hope to find out that the basic musical concepts that are embedded in the instrument can be absorbed by a group of children which, until now, was perceived as too young for playing or learning music. It is also my intention to analyse the different play patterns that have evolved from interacting with the playpen and to study how they can be employed in the performance of traditional musical instruments'. From a technical point of view, I expect to improve the Playpen's sensing techniques in order to be able to more accurately sense a the trajectory of delicate movement's. Initial experiments were conducted using commercial 3-D accelerometers and pressure sensors. I hope that these sensors will allow for better motion analysis that will lead to the development of more interesting mappings. Hopefully, this will allow for deeper bottom-up and top-down musical explorations and extend the instrument's play value. 135 The Musical Playpen: At: Immersive Digital Musical Instrument
Acknowledgements This research was done in the framework of my thesis work [7] at MIT's Media Laboratory Hyperinstruments group [8]. I would like to thank Seum- Lim Gan for his technical help and Professor Tod Machover for his support. References 1. Piaget J. The principles of genetic epistemology. Basic Books, New York, 1972 2. Warner B. Orff-Schulwerk: applications for the classroom. Prentice-Hall, New Jersey, 1991 3. Findlay E. Rhythm and movement: applications of Dalcroze Eurythmics. Birch Tree Books, Auckland, New Zealand, 1999 4. Langer S. Philosophy in a new key. Harvard University Press, Cambridge, MA, 1942 5. Csikszentmihalyi M. Flow: the psychology of optimal experience. HarperCollins, New York, 1991 6. Lehrman P, Tully T. MIDI for professionals. Amsco Publications, New York, 1994 7. Weinberg G. Expressive digital musical instruments for children. Master Thesis. MIT Media Lab, Cambridge, MA, 1999 8. Machover T. Hypesrinstruments: a progress report., MIT Media Lab, Cambridge, MA, 1992 Correspondence to: Gil Weinberg, MIT Media Lab, E15-491, 20 Ames St., Cambridge MA 0213, USA. Emaih gili@media.mit.edu 136 G. Weinberg