Jam Master, a Music Composing Interface

Transcription

1 Jam Master, a Music Composing Interface Ernie Lin Patrick Wu M.A.Sc. Candidate in VLSI M.A.Sc. Candidate in Comm. Electrical & Computer Engineering Electrical & Computer Engineering University of British Columbia University of British Columbia Vancouver, BC, Canada Vancouver, BC, Canada erlin@interchg.ubc.ca pswu@interchg.ubc.ca ABSTRACT This paper presents the interface design of the Jam Master system, an interactive music composing device that interprets human body gestures as music variable settings. The system allows multiple number of players to participate in the music making process. This paper begins with background information describing the complexity of music composition by musicians, the history of synthetic music, and the existing technologies that make a music system such as Jam Master possible. Then it continues with the discussion of the various key characteristics and requirements in real-time interactive multi-user system deign and a look at the similar systems that have been built. How the hardware systems have been selected and how the interfacing between the Polhemus sensors and Jmax (the software) are described. Then comes the player interaction section that describes how the Polhemus data is used to compose music, with the possibilities of varying the volume, pitch/frequency, and different sound effects. Finally the paper concludes with the testing and future implementation sections. Keywords Interactive music, interactions, multi-user, gesture recognition, interface design, real-time performance systems 1 INTRODUCTION Recently, human interface technologies have been receiving a great deal of attention from researchers from various disciplines including the engineering, computer science, performing arts and music areas. Interfaces, serving as the link between humans and computer or electronics, interpret human gestures and signals and process these cues to generate desired outcomes. In the lately published journals, interfaces examined mainly cover visual and audio systems that exploits the characteristics of human eyes, ears and speech, motion tracking systems that follow and react to human motions, as well as a great variety of input devices designed to replace pens and keyboards such as light pens, touch typing gloves, mice, and multi-dimensional joysticks. The Jam Master music composing system is intended to be run in real time, allowing players to generate music by moving their arms and tapping their bodies. Players can use the system to create pieces of music and learn about the different aspects underlying music composition. Overview The rest of the paper is organized as follows: Background describes music composition, history, and existing technologies, User Interface Design discusses the aspects of real-time, interactive, and multi-user systems that must be considered, Comparison of Existing Systems gives a brief overview of what other similar systems are out there, Interfacing the Hardware describes the hardware selection and the hardware interfacing process, Player Interaction explains how the data received in hardware is used to make music, and Testing, Future Considerations close off the paper with a look into what can be extended from this project. 2 BACKGROUND The background information for the Jam Master system can be separated into the music composition, history, and existing technologies sections. Music Composition From centuries ago, people have desires to make music for amusement purposes. By playing music instruments such as percussions, violins or mandolins, or using a part or parts of their bodies to whistle, clap, or sing, millions of pieces of melodies are composed everyday, and some become more famous than others. Musicians acquire music developing skills through formal training. At the fundamental level, rhythmic, melodic, textural, and harmonic processes of music and how these processes create structures in a variety of styles are studied.

2 Notation and aural recognition of rhythmic and pitch patterns, and compositional and analytical skills are vital in musicianship. Most importantly, much of the musician's craft is gained unconsciously through music composing experiences and training. And there is no simple answer as to how the knowledge, skills, experience and gestures are interrelated in music making [1]. Typically, the general public has neither the special knowledge in computing nor the specific music training to make music. On the other hand a musician would like to have maximum control over the music being played. Thus, a music composing system that targets the broad public and musician as its players should have the flexibilities of allowing experienced users to alter the music as much as possible while giving those with little or none music background and training the chance to be participants. Such a system would open up new ways to musical creativity, and musical education, regardless of the players prior musical knowledge [2]. History As technology advances, various synthesizing methods make it possible to generate music from electrical signals. At the early stages of creating electronic music, people were mainly concerned about imitating the sounds coming from the existing music instruments, thus the electronic keyboards or the electronic drum pads. These electronic instruments are not meant to replace the conventional ones, as the sounds generated, although carefully reproduced, are never close enough to the original ones. The earliest interactive computer music systems include Groove, designed by M. Mathews and his team at AT&T Bell Laboratories around 1968, and A Computer Aid for Musical Composers, developed at the National Research Council (NRC) of Canada around 1970[1]. Both systems have the basic music describing, manipulating, playing and storing functions in terms of real time interaction. As Mathews pointed out the main characteristics of music composing systems, The desired relationship between the performer and the computer is not that between a player and his instrument, but rather that between the conductor and his orchestra [3], it is clear that these systems are not just different pieces of apparatuses but ensembles that are not limited to one certain type of music. Existing Technologies The birth of Musical Instrument Digital Interface (MIDI) led to a new era in music synthesis. MIDI is a standard agreed upon by the music industries in 1983 for digitally synthesized music. It s designed to represent the sounds of certain music apparatuses such as keyboard and wind instruments in the digital format, and can be used to exchange music data between synthesizers, sequencers, computers, interfaces, and controllers thru a daisy-chained serial bus [4]. The standardization is seen as a big step in the computerized music business, and MIDI becomes widely used for musical representations. Before the idea of virtual reality (VR) came into play, some operators of electronic keyboards and guitars expressed their resentments towards having to actually hold devices in order to play. Therefore when VR becomes popular, the physical interfaces (instruments) are taken out of the music controller systems in some designs. Instead, cyber gloves, polhemus sensors, or other wearable sensors are used to measure the players music gestures while performing with an air guitar, an air drum, or an air harpoon, to determine the sounds to be generated. To extend the VR idea further, developments in music controllers surge as inventors allow music composing to be of free form, not limited to the existing music instruments. Through interacting with the human bodies, music controllers can generate pleasing music with the help of computers. 3 USER INTERFACE DESIGN The user interface for Jam Master is to be designed with the following characteristics that apply to real time, interactive, and multi-player systems in general. Technical Considerations for Real Time Systems The requirements for a music interface would vary from systems to systems, depending on what the intended achievement is and where the system will be used. Pennycook illustrates the timing requirements in a hierarchical fashion for real time systems as in Figure 1[1].

3 The first level deals with controlling the formal music structures, where the instrumentation and number of input channels are considered. Then the input data is collected from the sensor devices used at the performance execution rates level. At the third level, performance and synthesis data are merged into a continuous stream and the stream is sent to the synthesizer as updates. The synthesizer then computes the sound samples to be outputted, according to the hardware instruction rates. And finally at the last level, the sound samples are played using the audio conversion subsystems. Since the system is intended to be run in real time, the figure also lists the desirable timing constraints at each level. As the computer music system is classified as a soft real time system, it is not critical for each level to meet the timing constraints as absolute deadlines. If the constraints cannot be met, the music output would be somewhat distorted, yet not to the situation that it becomes complete noises. Features for Interaction There are some key characteristics that an interactive system should include at the design phase. Borcher, in his paper for the WorldBeat system, described several aspects of what an interactive system looks like [2]. First of all, the system should be consistent and explorable. Players should be able to access all the functions that the system provides without having the switch between input devices. Also, the system should allow players to actively participate and navigate the different features of the system, and thus produce innovative results on their own. Secondly, the system would preferably be intuitive and non-technical. Making the system easy to use will encourage people with little music experience to operate without having to do lots of practices. In an interactive environment, it would be difficult to require the participants to have the same music or system-wise training. Therefore people with different abilities should all be able to use the system. Furthermore, since the system is designed to be interactive, it shouldn t require players to follow a particular set of instructions in order to generate outputs, and thus the non-technical issue. Thirdly, the system should be cooperative. To extend the concept of interaction, making the system available for multi-player use adds another variable to the system. While the system can be operated by a single player alone, the addition of other players enables cooperative learning of music creation. And it would probably make the system more enjoyable. Weighting between the Main Composer and Side Players In a multi-player system, generally one leading role would be assumed. This is to make the decision making process somewhat simpler and to generate a more coherent result than weighting all participants as equals. However, the limited opportunities for interaction given to the side players create drawbacks for them, such as having little to distinguish the use of the system from listening to a piece of music made by someone else, and having a very limited sense of involvement during the composition. Therefore, the factor that is used to weight the activities by the main composer and the side players must be carefully decided. 4 COMPARISON OF EXISTING SYSTEMS There are several devices that have some resemblances at the fundamental level to what the authors plan to implement. Predefined Music Library The Digital Baton, built at the MIT Media Lab, incorporates optical trackers, pressure sensors and accelerometers to track operators hand gestures [4]. With that information, operators by moving the baton with certain beats or in large gestures can act as conductors to make changes in transitions in sound groupings or underlying rhythms. This digital baton is similar to the Jam Master in that it contains a predefined music library. From the library, the synthesizer of the Digital Baton selects and plays pieces of music. In Jam Master, operators will be able to choose different sound effects from the library to play as they see fit. Wearable Device Building a wearable music interface is another important concept in the Jam Master design. Maggie Orth and Rehmi Post at Media Lab built a musical denim jacket that is equipped with a 12 key keypad and a synthesizer [4]. Operators would press the keypad to generate the desired music with the jacket. In another project titled dance sneakers, sensors are built into the shoe and data is offloaded to the synthesizer over a wireless link. In Jam Master, operators will have to wear the sensors on their bodies so that the body gesturing signals can be sent to the computer for further processing. The wearable aspect gives operators mobility within some range, which allows the players to have a certain degree of freedom in movements. 5 INTERFACING THE HARDWARE In order to measure the body gestures, a piece of hardware device must be chosen to receive the inputs of the body signals. Then, an interface for the chosen hardware must be build so that data from the hardware will be ready to be accessed by the music composing interface. Hardware Selection There are several aspects that we can take advantages of in modifying music, including volume, pitch, frequency, and etc.. And these aspects can distribute over a range of values. Therefore an input device with multi-dimension measurements must be used for the system.

4 The Polhemus Fastrak sensors, provideing dynamic, real time six degree-of-freedom measurement of position (X, Y, and Z Cartesian coordinates) and orientation (yaw, pitch, and roll) are therefore the most promising electromagnetic tracking system that is available for the authors to use. Furthermore, each Polhemus system can accept data from up to four receivers at an reportedly update rate of 120 Hz (with a single receiver) and a remarkable 4ms latency [5]. The received data is transmitted to the serial port of a computer for further processing. This allows the Jam Master system to have up to four players composing music at the same time. Since no line of sight is required between the sensors and the receiving base, players of the Jam Master system wouldn t need to worry about their bodies blocking the receiver. For selecting music from the sound library, the icube system will be used. icube can be connected with various types of touch sensors that will be used for the sound effect selection. Interfacing the Polhemus and Jmax Jmax, the signal processing software that the authors have chosen for music composition (see the next section), uses patches as building blocks in the construction. Therefore, it is necessary to make an interface that outputs the measurements from the Polhemus in Jmax. The patcher for the icube system is written by a different group that is using the Jmax software as well, and therefore is not described here. Each polhemus sensor has six measurements, and therefore four sensors will output a total of twenty-four values. Using the existing driver for the Polhemus, a Jmax patch would be able to access the values. Under a hierarchical structure, a patcher that outputs the twenty-four values of the Polhemus is illustrated in Figure 2. Figure 2: The Interface for the Polhemus Fastrak System. In Figure 2, metro is a patcher and will generate continuous bangs at its output periodically once it receives a bang at its inlet. The Mypolhemustop patcher receives these continuous bangs and calls the driver that reads in the data from the Polhemus. Mypolhemustop would then format the data into four arrays, one for the data received at each sensor, and output them to its four outlets. Mypolsensor, once receives the data array containing the six readings from a sensor, will parse the arrays and output each value to one of its six outlets. 6 THE JMAX PROGRAMMING ENVIRONMENT The programming environment that was used to develop Jam Master is called jmax. It is an object-oriented, graphical programming environment based on the Max/FTS core with a java-based GUI. Objects (called patchers) are implemented in one of two ways: a) using the GUI to construct new patchers out of existing sub-patchers graphically, by using lines to connect sub-patchers together; or b) writing C code to implement the function of the patcher, then using the C as a basis for developing a jmax control object. jmax objects each have a set of inlets and a set of outlets for data. The inlets are the input ports to the patcher, and conversely, the outlets are the output ports. There are also 5 basic types of data: integer, float, message, Boolean and bang. The bang data type represents a type somewhat analogous to a switch; it is used as a signaling type for events. jmax objects are interconnected by lines in the graphical interface. These lines represent conduits for data messages to flow from an outlet of one object to the inlet of another object. Signal processing in jmax is accomplished by using jmax signal objects. Signal objects implement various signal processing constructs, like filters and delay lines. There is also a signal data type which these constructs operate on. Signals can be filtered, added, subtracted, multiplied, divided, and they can also be fed into a digital-analog converter (DAC) object at which point the signal it represents gets converted into sound that we can then hear. The best feature of jmax (at least in the authors opinions) is that recompilation is not necessary every time you modify a patcher, since the objects themselves are not changing, only the interconnection between objects. Once you make a change, to see it work involves merely changing to a different editor mode in which your patcher can be executed immediately. 7 PLAYER INTERACTIONS Jam Master requires up to four people to operate. One person is the main composer, he decides what notes are to be played and the volume of each note. The other players contribute variations of the composer s note to the symphony. The other players modify the composer s note by making it shifted slightly in frequency

5 and/or changing its amplitude. Each of the other players can also add sounds from a sound effect library to the symphony. In this way, the overall music is influenced by each of the players individual tastes. The players also need not be musically inclined; with some practice, the output sounds created could be an expression of their collective creativity. Volume/Amplitude Adjustment Each player controls his/her note independently of the other players. Each players Polhemus sensor data controls his/her own amplitude multiplication factor. The sensor data is normalized to a value between 0.0 and 1.0, along a linear scale. The note is multiplied by this factor to change its amplitude (where 0.0 means the note is muted and 1.0 means the note volume does not change). To produce this normalized factor, the Polhemus sensor data needs to be processed. The (x, y, z) coordinates returned by the sensor are used to calculate the distance between the player and the Polhemus transmitter. This distance is then normalized to between 0.0 and 1.0 by dividing this value by a normalizing factor which represents the maximum separation distance between the sensor and the transmitter. This need not be the actual maximum possible distance that the sensor is capable of reading; it could just be an arm s length or so. note the composer has generated). Pitch adjustment is accomplished by converting the main note to a frequency in Hertz, then adding a frequency offset (which may be positive or negative) and playing the resulting sound. It is the frequency offset which is under control of the other players. The frequency offset is computed based on the Polhemus sensor data received. The quantity of importance for this purpose is the roll component of the (roll, pitch, yaw) data recorded by the sensor. Roll is measured in a range from +180 degrees to 180 degrees. We have chosen to not normalize this data before applying it to the note. However, normalization would not be difficult to implement in the future. Since the roll varies from +180 to 180 degrees, the amount by which the frequency can be changed varies from +180 to 180 Hertz; that is, the amount of roll is added to the frequency of the note and played. Figure 3: Volume Adjusting patcher (note: Polhemus patcher detail has been abstracted away for simplicity) Pitch / Frequency Adjustment As with volume, pitch is also controlled independently by each player. Each player (other than the composer player) generates a slightly off-pitch version of the main note (the Figure 4: Pitch Adjusting patcher (note: Polhemus patcher detail has been abstracted away for simplicity) Sound Effects Each player also has at his/her disposal a library of sound effects that can be loaded in when the application starts. How the sound effects are activated is simple. Each player wears a set of touch sensors on his/her body. The sensors are connected to an icube that allows the signals to be interfaced with jmax. Upon touching a sensor, jmax sends a bang signal to play back the corresponding sound file.

6 Figure 5: Sound Effects Playback patcher Combining each Players Notes Combining notes is also simple. Each player s notes are reduced in amplitude by a factor of four. Then the signals are summed and fed into an oscillator and finally output via the DAC. The figure below shows how this was done in jmax (figure 6). Figure 6: Combining Player Notes patcher Some consideration was given to the issue of whether the composer should have the ability to adjust the weight of his component of the music so that he could make himself sound more dominant or less dominant over the other players. We have decided to leave this feature for a future revision of Jam Master since it is fairly trivial to implement and does not contribute greatly to the overall description of the interface to Jam Master. 8 TESTING Informal testing was completed on the application. The application was broken into its constituent parts and each part s operation was verified individually. The Polhemus sensor was also tested during the development of the polhemus patcher in jmax. The polhemus patcher was verified simply by executing the patcher in jmax and observing the values at the patcher outlets as the sensor was moved around in space. Position in (x, y, z) coordinates was displayed and verified, along with the (roll, pitch, yaw) coordinate outputs. We have not yet tried testing the application as a whole, and as such we have not tried the application with more than one player. It is also difficult to test because to use the application in such a way that produces harmonious music, a lot of practice and/or musical aptitude is needed. We (the authors) do not have this kind of experience. The best the authors have done is generated what sounds more or less like volume-varied random noise. The intended use for this application involves the composer playing a tune (perhaps something classical), with the other players creating accompaniment sounds. When done correctly (and with enough practice), the end result should be very harmonious. Problems Encountered Over the course of this project we had many difficulties which prevented us from achieving our goal as fast as we would have liked to. One of these problems had to do with the sound card installed in the computer. Our access to the sound card was very sporadic. This was because our read and write access to the device in Linux changed constantly, possibly because of the other users tweaking. In order for jmax to work (sounds and music can be produced and heard) the read and write permission for the sound card must be set correctly. Another problem we had was with the Polhemus sensor. The sensor is read via the serial port on the computer. However, for reasons not known, the read and write permission to the device was also sporadic. The permissions for the serial port also need to be set properly for the Polhemus to function. We also seemed to have some trouble with creating jmax objects from existing C source code. We felt this was because of the inadequate documentation included with jmax. However, the jmax mailing list proved to be of much assistance while trying to figure out some of the more obscure things in jmax. Also on the subject of jmax objects and poor documentation, we felt that even creating patchers from existing sub-patchers was difficult because some of the sub-patchers lacked online help (or any help at all for that

7 matter). For these sub-patchers, either trial-and-error was used to figure out the sub-patcher (its inlet/outlet types, and what each inlet/outlet represented), or the sub-patcher was worked around completely by implementing the function in a different way. 9 FUTURE EXTENSIONS / FUTURE WORK We would like to add features to the application in the future, such as echo/reverb support, support for karaokeing, as well as more signal processing features, such as filtering. We feel it would be interesting to add support for even more players to the application. More players and more features mean more variables, and we believe that having more variables available to the player increases the opportunity to express creativity. We think it would be interesting to add a module for beat detection and beat modification. For example, one idea was that the players could form circles and that the radius of the circle determines the beat of the music being played (this implies that the beat is not controlled by the composer; think of it as a drum machine type of feature the composer follows the tempo of the drum beats when he/she is composing) Another interesting thing might be a visual type of display that is modulated by the players positions and/or music being played. Our idea is somewhat similar to the output visualization plugins popular in most mainstream MP3 player programs, such as Nullsoft s Winamp. We mentioned above that each note generated by the players and the composer is given equal weight when they are combined. It would be interesting to give the composer control of this weighting so that players could be emphasized and de-emphasized as the composer saw fit. Finally, we think that adding MIDI instrument controls would be a good idea, by giving the composer greater freedom in terms of what instruments he/she has available for use. If this can be done without entering numbers into a keypad (perhaps by touch sensors worn by the composer), this adds yet another dimension of creativity that can be expressed. 10 INFORMATION AND QUESTIONS For more information, contact ssfels@ece.ubc.ca. ACKNOWLEDGEMENTS The authors would like to thank Prof. S. Fels, the members of his lab as well as several members on the jmax mailing list for the technical support given. REFERENCES 1. Pennycook, B. Computer-Music Interfaces: A Survey. Computer Surveys, Vol. 17, No. 2, June 1985, Borchers, J, WorldBeat: Designing a Baton-Based Interface for an Interactive Music Exhibit. ACM CHI 97 (Atlanta GA, March 1997). ACM Press, Mattews, M. GROOVE, a program for realtime control of a sound synthesizer by a computer, Proceedings of the 4 th Annual Conference of the American Society of University Composers. ASUC, Paradiso, J. Electronic Music: New Ways to Play, IEEE Spectrum Select, 5. Polhemus Fastrak at

8