Trombosonic: Designing and Exploring a New Interface for Musical Expression in Music and Non-Music Domains

Transcription

1 Trombosonic: Designing and Exploring a New Interface for Musical Expression in Music and Non-Music Domains Oliver Hödl and Geraldine Fitzpatrick Human Computer Interaction Group Institute for Design and Assessment of Technology Vienna University of Technology oliver@igw.tuwien.ac.at, geraldine.fitzpatrick@tuwien.ac.at Abstract The Trombosonic is a new digital music instrument based on the foundational principles of the slide trombone. An ultrasonic sensor combined with a red laser allows the performer to play the instrument using similar movements to playing a trombone to change the pitch, by moving one hand back and forth even though there is no physical slider available. Furthermore, additional sensors enhance musical expression by gestural movement of the whole interface and by using the breath. Due to its compact size and the lack of a slider, the Trombosonic can be played in many different ways. This inspired us to do an informal evaluation to explore the potential applicability of our prototype in different fields and settings other than music. We identified a certain suitability for old and young people and a new possibility for people with restricted mobility to play such a musical instrument. Further development might include a builtin microphone to use the human voice and an expansion of the synthesizer s features. Keywords-Sound and Music Computing, Interface for Musical Expression, Exploratory Evaluation I. INTRODUCTION Musical instruments equipped with sensor-technologies allow many different ways of expression and interaction [1]. Apart from using them for musical purposes the application of such versatile interfaces can be manifold (e.g., [2], [3]). In this paper, we present a new multi-purpose interface named Trombosonic. The primary intention was a new digital music instrument inspired by the slide trombone. Hence, we started to design the interface under some self-defined constraints. Unlike many existing approaches, we did not augment a trombone (e.g., [4]) or used the instrument as an example for a digital music interface imitating the trombone s look and feel to create an electronic slide trombone (e.g., [5]). For our development we rather took the technique for playing the trombone as a guiding principle only, to enable an embodied control of sound with a preferably simple and compact hand-held interface. Our initial design considerations led to preconditions which address sensor-technology and construction. To balance functionality and complexity of the interface and keep it as simple and cheap as possible, we decided to use only standard offthe-shelf low-cost hardware, such as an ultrasonic sensor, push buttons and an accelerometer, to mention some of the important ones. By doing so, we could keep the costs for hardware and material below 100 Euros in total and were still able to explore a range of different sensors within one device. Figure 1: The final Trombosonic prototype Throughout the design process, the basic intentions regarding appearance, functionality and materials changed significantly. We initially started with a paper-made tubular prototype to simulate a trombone. The final interface is shown in Fig. 1 and illustrates the visual difference to a traditional slide trombone (see Fig. 2). Most notable is the missing typical slide that characterises a trombone. Despite that, it is played like a trombone with slide motions by holding it in one hand, either left or right, and changing the pitch by moving the other hand back and forth. The name Trombosonic is the combination of the two words trombone and sonic. However, during the development phase it turned out that the device might also be useful for other applications as a handheld interface. Apart from its original purpose to serve as a musical instrument, an exploratory evaluation has shown its potential applicability in fields such as education, sonification, therapeutic prevention and rehabilitation. The main contribution of this paper is the presentation of the new digital music instrument Trombosonic along with an exploratory evaluation. We use both expert knowledge and the experience of extreme users [6] to identify potential future applicability in music and non-music domains. In the following we start with the description of similar research and existing literature our project is built on. Then we go on to describe the design and the functionality of the Trombosonic. Finally, we present the exploratory evaluation 54

2 Figure 2: The jazz trombonist Roman Sladek plays a traditional slide trombone (Photographer OhWeh) Figure 3: Playing the Trombosonic, the red laser in the palm indicates the direction of the ultrasonic sensor that shows the potential applicability of our prototype. II. RELATED WORK Both researchers and artists have used the trombone for their work in many ways. Composers appreciate the trombone as very adaptable system for capturing, suspending and altering shards of sound using different electronic extensions to create new sounds and sample external sources [7]. Farell augmented a trombone by using a minimal-hardware ultrasonic sensor for the slide, a modified mouthpiece and a loudspeaker in the trombone bell to change the original sound of the trombone for his electro-acoustic performances [4]. A very simple prototype using an optical sensor to detect the position of the slide was created by Lemouton et al. to realise a gestural interface for a traditional trombone [8]. Instead of augmenting existing instruments, Bromwich built a completely new instrument, the Metabone, using only the trombone s dynamics and characteristics [9]. Su et al. present an electronic trombone for the entertainment of children and a playful introduction in musical instruments [10]. The Double Slide Controller derives from the traditional trombone [5]. It looks different though and appears as a complex interface. Unlike the presented examples that use the trombone as a model or augment an existing instrument, we wanted to combine its most promising features within one simple and compact interface. Keeping in mind the trombone as original instrument and its possibility to create sound by a unique hand gesture, we also wanted to provide new features and embodied interaction that goes beyond the usual musical purposes. III. OVERALL CONCEPT The Trombosonic s hand-held interface is purely electronic without any loose or moveable parts. It is held in one hand, either left or right, with a pinch grip. For data processing it uses an Arduino Duemilanove microprocessor [11]. An attached RedFly WiFi-Shield [12] sends sensor data as OSC messages wirelessly to a computer running Max/MSP for sound synthesis in our particular case or any other OSCcompliant musical application. The casing of the interface is cylindrical with rounded ends and made of polystyrene and wood (see Fig. 1 and Fig. 4). This keeps it lightweight but stable and handy. All electronic devices are bolt-on or glued. Additionally, four aluminium rods provide a good grip and they round out the overall appearance. Its total weight including batteries is 294 g (10.37 oz). For powering the Arduino, a battery pack is included at the bottom of the interface which holds four standard AA batteries. The longest period of time that the Trombosonic was turned on for testing purposes was 130 minutes and no energy problems were observed during this time. An accurate test regarding energy consumption has not been done yet. Both the compact design and the wireless network communication ensure free and easy movement during usage within the range of the arm and without being wired to the computer. The whole set of sensors and why they are specifically used to enable embodied musical expression, are described in detail in the following. IV. MUSICAL EXPRESSION The Trombosonic s primary intention is to serve as a musical instrument. Hence, it has several features that enable expressive sound control, including (1) four push buttons and four LEDs, (2) an ultrasonic sensor, (3) a red laser pointer, (4) a thermal resistor, and (5) an accelerometer. For its use as a musical instrument the sensors and actuators are configured to work for particular musical purposes. All features are shown in Fig. 4 as they are located on the device and an overview with a short description is given in Table I. A. Physical sound generation Four push buttons, mounted on the top board, enable the control of the basic functions. They are ordered in a square and operated with the middle finger and the ring finger. This allows a good grip using the other two fingers while pushing the four buttons. For additional visual feedback each button is connected to an LED in a different colour, which flashes when the button is pushed. All combinations of how the buttons can be pushed and the corresponding functions are shown in Table II. The Trombosonic uses a subtractive sound synthesis. The default frequency of the oscillator is 440 Hz. Button 1 turns on the sound, while button 2 allows the user to save and hold the actual frequency which changes continuously according to play. With this function, the player is able to explore the acoustic frequency spectrum endlessly or at least within 55

3 Figure 4: Description of all features of the interface TABLE I: Overview of features and functionality Feature Four buttons Ultrasonic sensor Red laser pointer Accelerometer Thermal resistor Functionality Tone on/off, set synthesizer param. Pitch/frequency control Direction of the ultrasonic sensor Position/movement of the interface Using the player s breath the human acoustic range. Buttons 3 and 4 switch between waveforms of the oscillator and a Low Frequency Oscillator (LFO). Pushing buttons 1 and 2 or 3 and 4 together switches between filter types and turns on the laser. Pushing all buttons at once, resets the reference value of the thermal resistor. All functions are described later in detail. B. Embodied expression The design of Trombosonic enables a range of embodied expressions in play. The ultrasonic sensor [13] at the front enables the typical pitch control of the generated tone as known from the slide trombone. Unlike the traditional instrument the Trombosonic has no slider or handle. Instead, the red laser pointer (a disassembled off-the-shelf model for presentations) indicates the direction of the ultrasonic sensor for a better orientation of the pitch-steering hand as shown in Fig. 3. While moving it back and forth a red dot is projected on the palm. This realisation allows the player to play the instrument with two hands, comparable to a slide trombone which also makes it familiar to spectators in its embodied movements. Because the ultrasonic waves can bounce off any object, TABLE II: Summary of button functions Buttons pushed at once Functionality 1 Tone on/off 2 Set new frequency 3 Switch oscillator wave 4 Switch LFO waveform Switch filter type Laser on/off Set thermal resistor value other button combinations not used yet the second hand is not mandatory, thus the Trombosonic can also be played with just one hand and interact with other objects. These objects may be items within the performer s environment, or the body itself. Whatever interface is pointed on, the distance is transformed into sound. Even spectators who are moving or waving hands can allow interactive sonification of both performer and audience. The laser pointer can also be turned off and on at any time during a performance to avoid dazzling the spectators. Another embodied sound control is realised with an accelerometer [14] that measures the interface s movement in three dimensions. The actual synthesizer implementation uses two of them. The device can be turned around the longest axis (the one the red laser points to) and up- and downwards to control frequencies of the LFO and the filter. Given that the trombone, the source of our inspiration, is a wind instrument, we also included a mouth piece in our interface. Unlike the slide trombone, it is for additional expression only and not the origin of the tone. For reasons of simplicity we did not use a complex breath analyser [4], [5] but a simple thermal resistor [15] to recognise the player s breath. During the design process we used this value to intensify different parameters of the synthesizer, such as the bandwidth of the frequency filter. However, with the actual prototype, the breath control gives the volume a boost as this seems to be comparable with a traditional wind instrument. C. Sound synthesis For our applications the Trombosonic uses Max/MSP for a subtractive sound synthesis. The whole patch is controlled remotely with the wireless interface and receives nine different sensor-values (see Table I). To generate a sound, the player can choose among simple waveforms which are attenuated by an ADSR (Attack Decay Sustain Release) envelope, an LFO and filter effects. Certain parameters can be controlled in real-time with the Trombosonic s hand-held interface. For this prototype we focused mainly on the interface and minimised the synthesizer s features. Hence, the sound reminds a little of old synthesizers. Furthermore, there is no special musical training or knowledge needed to play the Trombosonic and to explore its features. V. EXPLORATORY EVALUATION We did an informal evaluation of the Trombosonic as a musical instrument and explored the potential applicability of our prototype in different fields. For this purpose we asked experts at a competition for new musical instruments and a researcher in game and interaction design. Additionally we build on knowledge from existing literature about the value of extreme users [6]. The positive impact of music and the suitability of musical instruments in various non-music domains have already been shown. For instance they can be used as playful, toy-like devices for nonspecialists (e.g. [3], [16]) or for therapeutic prevention and rehabilitation (e.g. [17], [18]). This inspired us not only to use expert knowledge to evaluate the Trombosonic but also to give it away to people with different abilities and ages such as a 92- year-old woman and a 13-year-old boy. We considered them as untypical users for new musical interfaces and expected them to help to explore the Trombosonic s potential beyond performances. Everyone participating in the exploratory evaluation was not involved in the project before and saw the device for the 56

4 first time. After a short introduction they were allowed to play the interface freely. Afterwards they were asked to tell us about their experience. In addition, we took photographs of their explorations and took notes of their comments. All subjects started to play the Trombosonic with its general sonic and gestural features we described in section IV. With each of them we spent about minutes for exploration and talked with them about their experience. For some we slightly changed single features tailored to their anticipated interests and needs, as will be described below. A. New interface for musical expression - expert evaluation To hand out the Trombosonic to a musician is the most obvious test for a musical instrument. We did so in early spring 2013 for a performance in Vienna where the Trombosonic was used as special instrument for a certain part of a show. The artist used it as a solo instrument during one song. However, to fit better with the other unusual users and to get the most interesting and diverse results, we did something different. We applied for the annual Margaret Guthman Musical Instrument Competition [19] which is considered one of the largest competitions for new musical instruments. The Trombosonic was chosen out of more than 70 submissions to take part in a performance as one of 17 semifinalists which means an acceptance rate of lower than 25%. The successful submission to this highly competitive and renowned competition proves that the Trombosonic is already wellregarded as a new musical instrument. The actual performance took place in Atlanta, USA, in April We took advantage of this event to get the official feedback of the expert jury as well as the opinion of other participants and audience members when presenting it as a new musical instrument. The performance at the competition was successful and two pieces were presented: One original electro-acoustic composition and one rather mainstream oriented piece accompanied by pre-recorded playback. People in the audience as well as the jury enjoyed the presentation of the many different features and how the Trombosonic was played in a trombone-like manner during the first piece. The second piece was called Trombopolka and was intended to be a tribute to the original instrument, the slide trombone. The Polka is a popular genre of folk music. Some audience members explicitly stated after the performance how they liked the combination of traditional music and the new musical instrument. The experts mainly criticized deficiencies in the sound synthesis and some spectators missed the acoustic traceability of the breath sensor. One suggestion from another musician was to integrate a microphone for additional sound creation using the human voice. Two other performers pointed out the compact and wireless design which makes it easy for embodied performances as they anticipated. In summary, the performance at the musical instrument competition confirms the potential of the Trombosonic as a new interface for musical expression and various comments from new musical instrument experts suggest the direction of future revisions and improvements. B. Physical training for older adults We then gave the Trombosonic to a 92-year-old woman who is a relative of one of our project members. She was willing to help us for the evaluation during a visit at her own house. She has full mental abilities apart from some forgetfulness from time to time, as she confessed herself. She Figure 5: A 92-year-old woman playing the Trombosonic: First impression (left), standing to operate it in another way (right) is still able to walk without a cane in her home. She told us, she uses a walking stick only outside as a precaution and especially during the winter season. However, according to her own description her movement abilities are getting worse and her visibility is already in a bad condition. Asked for her musical knowledge she said, she had learned to play the piano a long time ago and loved to play music and to sing. Now she is unable to play any more since she cannot see the keys and the score. We did not present the Trombosonic as a music instrument to her. According to what literature suggests in relation to physical activity and elderly people [20] we rather said it was an acoustic training device. Addressing her own musical experience, we changed the original electronic sound with a piano synthesizer to make it sound more familiar to her. After an explanation of the buttons and some possibilities to make sound, she started to handle it by herself. Conversation with her and our own observation have shown that the originally intended way to play the Trombosonic with two hands was not very convenient for her. What was notable though, was her behaviour changing her hands holding the device alternately in both hands and finally she even stood up to operate it in another way as she noted (see Fig. 5, right). She said she tried to find a good way to hold it and at the same time preventing her arms from getting tired when moving the device by changing hands. Unlike all other participants of the exploratory evaluation, she was the only one considering tiring issues during playing the instrument. This might be important when using the Trombosonic for older adults or rehabilitation. It appeared to us that she mainly concentrated on the device itself instead of producing particular sounds. However, at the end of our session she summarized her experience: I really enjoyed making it sound like a piano doing moves I am usually not used to do. Though I do not know how it works and why it sounded like a piano (Translated from German). Overall, we identified a certain interest in the Trombosonic and her different ways to handle it. Following Rolland et al. who illustrate that regular physical activity is a key component of successful aging [20] and Bruhn and Schröter who discuss the positive impact of making music in old age, we propose the Trombosonic as a potential device for elderly 57

5 people. It might be a good way to combine physical and musical activity. C. Playful interface for children When talking about musical play and young children, Tarnowski explains functional musical play might include exploring vocal, instrumental, and environmental sounds as well as the way in which these sounds are made [21]. This motivated us to give the Trombosonic to a young boy aged thirteen (Fig. 6, right) who was visiting our lab for a trial internship. He has no instrumental training but considers himself a very interested listener to music which is also indicated by the big headphones he wears around his neck all the time. Additionally he started to make music with his computer a little while ago, experimenting with a softwaresynthesizer. Similar to the older adult, we explained the basic functionality of the Trombosonic to him and how to handle it. When he started playing we observed, most different to all other evaluation participants, that he really seemed to focus on the music. We also noticed that he played the Trombosonic mostly in its originally intended way using two hands. However, once he started to roll the interface with one hand on the table to create a smooth wave-like sound using the accelerometer. He was the only one who used the movement features of the interface in this physical way together with other objects such as the table. In all, the young boy carefully analysed the different features and ways to play the Trombosonic throughout his whole session. Following his own words it was a lot of fun and he would like to control his own sounds with the interface. We propose the Trombosonic as a suitable instrument for letting young people playfully explore music without being able to play a traditional instrument. D. Sonification and people with disabilities Finally, we asked a researcher with expertise in interaction design within our lab to tell us about his experience with the Trombosonic. After an initial explanation of the basic functionality we let him explore the device. It was significant that he started to use it as a one-handed device despite our initial advice to play it in a trombone-like manner. Following his own intuition (as he defined it by himself) he started to walk around the room using the Trombosonic as a sonification device. He started to explore the environment acoustically while pointing the device onto different walls and surfaces (Fig. 6, left). Furthermore, he turned the device around pointing the ultrasonic sensor towards his own body. Moving it back and forth he started explaining: Look, now it is a one-handed instrument. I can play a trombone without my second hand (Translated from German). During his test he complained about the lack of clear feedback when using the buttons to control the synthesizer. Since he was not familiar with the synthesizer s options this was really a problem when trying to intentionally switch between wave forms and filters as he said. The trial with the interaction designer suggests some usability improvements for a more intuitive handling. Furthermore, it might be worth considering the Trombosonic as a one-handed musical instrument keeping in mind that thousands of people with disabilities in the UK, and millions across the world, are excluded from music making [22]. The Trombosonic could be such an instrument to enable those Figure 6: People playing the Trombosonic, as a one-handed instrument acoustically exploring a shelf (left), trying its features played with two hands (right) people and people with restricted mobility in general to gain a trombone-like musical experience. It could also have potential for people with visual impairment as a way to playfully explore their physical environment. VI. DISCUSSION The exploratory evaluation was not meant to be comprehensive but to complement the main contribution of this paper, the presentation of a new musical instrument. It gave us a differentiated impression of how people play the Trombosonic from the perspectives of both experts (in new musical instruments and in interaction design) and extreme users (very old, young). Their considered feedback, as well as their unanticipated uses, pointed to potential applicability that might be worth considering and gave some initial directions for future development. Overall, people tried various different ways to handle the interface, such as using one-hand only or both hands and while standing, sitting or moving around. The actual usage to produce sound was ranging from playing music following scores in a traditional way to acoustically explore the environment. The approach of using an exploratory evaluation when testing new musical instruments turned out to be qualified. It was inspiring to use expert knowledge as well as to see unexpected behaviour of unusual users. We argue that our assumption to widen the range for non-obvious applications by doing an exploratory evaluation was verified. At least for the initial of a new musical instrument and a new interaction device this opened a set of unpredictable possibilities for improvements and new directions to focus on for future development. Compared to the initial described approaches which augment traditional trombones or create new interfaces on the basis of the original instrument, our strategy to create a compact device has its advantages as the exploratory evaluation has shown. Despite its different appearance, people considered the Trombosonic to be a trombone-like instrument. At least when it is played as intended which happened during the instrument competition in our particular case. Analogously people tend to play with the interface in unusual ways and they explore its features as soon as they do not think of it as a trombone-like instrument such as the interaction designer and the older adult. Thus, designing a new musical instrument under certain constraints and evaluate the prototype in an exploratory manner brought the anticipated insights in unexpected and unpredictable user behaviour. The combination of experts and extreme users helped to go beyond the usual applicability 58

6 of this musical interface in fields such as healthcare and education. VII. CONCLUSION The Trombosonic is a new instrument for musical expression that derives from the slide trombone. However, it does not imitate the slide trombone either visually or acoustically, rather the principles of this wind instrument serve as a design inspiration for the interactive gestures. Push buttons, an ultrasonic sensor and a red laser allow an embodied playing of the instrument similar to the slide trombone changing the pitch with one hand moving back and forth. Compared to a traditional slide trombone, the whole instrument s size is much smaller and the slider is completely missing. Furthermore, an accelerometer and a thermal resistor enable an additional embodied expression. Moving the whole interface enhances the musical possibilities compared to the traditional instrument, while the use of the player s breath retains a typical feature of wind instruments. Along with presenting the Trombosonic as a new interface for musical expression we did an exploratory evaluation looking for its potential as a musical instrument as well as in other fields. Hence, we successfully submitted a performance proposal to an international competition for new musical instruments and gave the instrument to a 92-year-old woman, a 13-year-old boy and a researcher in game and interaction design. This let us identify different issues and unexpected aspects to keep in mind for future improvements. All cases also indicate the Trombosonic s suitability for various musical purposes as well as non-music applications. VIII. FUTURE DIRECTIONS For advanced prototypes of the Trombosonic we plan to integrate a microphone for additional sound creation using the human voice. Furthermore, the synthesizer needs some revision regarding the sound and better mapping of sensor values to single parameters, along with a more intuitive button control. Beyond technical improvement addressing mainly musical features, the evaluation suggests to adapt and use the interface in other domains. It could be used as training device for elderly people addressing physical and musical health-relevant activity or it could let children intuitively explore sound generation without being trained to play a traditional music instrument. REFERENCES [1] C. Dobrian and D. Koppelman, The E in NIME: Musical Expression with New Computer Interfaces, in Proceedings of New Interfaces for Musical Expression, 2006, pp [2] C. Pacchetti and F. Mancini, Active music therapy in Parkinson s disease: an integrative method for motor and emotional rehabilitation, vol. 393, 2000, pp [3] D. Robson, Play!: Sound toys for non-musicians, Computer Music Journal, vol. 26, no. 3, 2002, pp [4] N. Farwell, Adapting the trombone: a suite of electro-acoustic interventions for the piece Rouse, in Proceedings of New Interfaces for Musical Expression, 2006, pp [5] T. Henriques, Double Slide Controller, in Proceedings of New Interfaces for Musical Expression, 2009, pp [6] L. Holmquist, User-driven innovation in the future applications lab, Extended abstracts of the 2004 conference on Human factors and computing systems - CHI 04, 2004, pp [7] N. Collins, Low Brass: The Evolution of Trombone-Propelled Electronics, Leonardo Music Journal, vol. 1, no. 1, 1991, pp [8] S. Lemouton, M. Stroppa, and B. Sluchin, Using the augmented trombone in I will not kiss your f. ing flag, in Proceedings of New Interfaces for Musical Expression, 2006, pp [9] M. A. Bromwich, The Metabone: An interactive sensory control mechanism for virtuoso trombone, in Proceedings of International Computer Music Conference, 1997, pp [10] M. Su, W. Lee, and S. Chen, Electronic trombone: an interactive tool to promote musical learning and performance creativity, in Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, 2011, pp [11] Arduino. [Online]. Available: (last access 24. January 2014) [12] RedFly WiFi-Shield. [Online]. Available: (last access 24. January 2014) [13] Ultrasonic sensor. [Online]. Available: (last access 24. January 2014) [14] Accelerometer ADXL335. [Online]. Available: (last access 24. January 2014) [15] Thermal resistor. [Online]. Available: (last access 24. January 2014) [16] S. Jordà, Digital Instruments and Players: Part I Efficiency and Apprenticeship, in Proceedings of New Interfaces for Musical Expression, 2004, pp [17] C. Plahl, Musiktherapie - Praxisfelder und Vorgehensweisen, in Musikpsychologie. Enzyklopädie, Rowohlts, 2009, pp [18] M. E. Clark, A. W. Lipe, and M. Bilbrey, Use of music to decrease aggressive behaviors in people with dementia. Journal of gerontological nursing, vol. 24, no. 7, Jul. 1998, pp [19] Margaret Guthman Musical Instrument Competition. [Online]. Available: (last access 24. January 2014) [20] Y. Rolland, G. Abellan van Kan, and B. Vellas, Physical activity and Alzheimer s disease: from prevention to therapeutic perspectives. Journal of the American Medical Directors Association, vol. 9, no. 6, Jul. 2008, pp [21] S. M. Tarnowski, Musical Play and Young Children, Music Educators Journal, vol. 86, no. 1, 1999, pp [22] The One-Handed Musical Instrument Trust. [Online]. Available: (last access 24. January 2014) 59