MUSIC NEUROTECHNOLOGY: FROM MUSIC OF THE SPHERES TO MUSIC OF THE HEMISPHERES

Transcription

1 Symmetry: Culture and Science Vol. 26, No. x, page_first-page_last, 2015 MUSIC NEUROTECHNOLOGY: FROM MUSIC OF THE SPHERES TO MUSIC OF THE HEMISPHERES Eduardo Reck Miranda Interdisciplinary Centre for Computer Music Research (ICCMR), Faculty of Arts and Humanities Plymouth University The House, Plymouth PL4 8AA United Kingdom Abstract: The emerging field of Music Neurotechnology combines musical research with other fields such as Artificial Intelligence, Bioengineering, Neurosciences, Symmetry and Medicine. Musicians have an extraordinary opportunity nowadays to develop new approaches to composition informed by our understanding of the brain, which would have been unthinkable a few years ago. This paper reports on the outcomes of two projects: one is concerned with the development of brain-computer music interfacing (BCMI) technology and the other is looking into developing methods to express brain information by means of music. A BCMI system allows a person to control musical devices by means of commands expressed by brain signals, which are detected by means of brain monitoring technology. We are interested in developing BCMI aimed at medical applications and people with special needs, in particular for people with severe physical disability. In the second half of the paper we introduce the inside story of Corpus Callosum, a piece for chamber orchestra composed through a new approach to musical composition using information from brain scans. 1. INTRODUCTION Imagine if you could play a musical instrument with signals detected directly from your brain. Would it be possible to generate music representing brain activity? What would the music of our brains sound like? These are some of the questions addressed by our research into Music Neurotechnology 1. This paper presents the outcomes of two projects that epitomize the research that we have been conducting it the field of Music Neurotechnology at Plymouth University s Interdisciplinary Centre for Computer Music Research (ICCMR): one is concerned with the development of brain-computer music interfacing (BCMI) technology and the other is looking into developing methods to represent and express brain information by means of music, literally and metaphorically. The work presented here in a way continues a 1 The term Music Neurotechnology appeared in print for the first time in 2009 in the editorial of Computer Music Journal, volume 33, number 1, page 1.

2 2 E. R. MIRANDA tradition in musical research that can be traced back to the Middle Ages, however with a contemporary outlook. Latin translations of enigmatic but allegedly illuminating texts from ancient Egypt and Greece formed the basis of the European scholastic understanding of the world from the late Middle Ages through to the Renaissance. Music had a privileged place in the socalled Hermetic Tradition that flourished at the time. It was believed that music evoked cosmic spirits that could influence our body and soul, and to a certain extent, social behavior and politics. Scholars looked into developing astrological music, using combinations of tones representing numerical relationships and properties associated with planets and stars (James 1993). They aspired to produce music that could establish sympathetic relationships between the human soul and cosmic forces, and which might even have medicinal powers. The belief that astrological music had the power to influence our body and mind stemmed from the assumption that there were symmetrical correspondences between the notions of music of the spheres (i.e., cosmic music), music of the human organism (i.e., harmonious and healthy functioning of our body) and musica instrumentalis (i.e., ordinary music played on musical instruments). Such symmetry provided support at the time for explanations of how the human organism reacts to music. For instance, if a piece of musica instrumentalis embedded numerical information that evoked planet Venus, then the person who heard it would have become (or remained) calm and peaceful, because this planet was associated with love, calmness, softness, and so on. Obviously, our understanding of the world has evolved beyond recognition since the Renaissance period and so has the notion of music. Paradoxically, however, those medieval scholars elicited a number of issues and ideas that are still pertinent today, but which can be tackled with far more sophisticated theoretic and scientific tools than before. For instance, take the notion of symmetrical correspondences. Symmetry is indeed an important theoretical scaffold to describe the world. But today we have a much more advanced understanding of the concept of Symmetry in nature than we had 500 years ago: the notion that symmetrical natural phenomena are prone to rupture constitutes one of the pillars of our modern understanding of the evolution of material structures and complex systems, such as the human brain. For instance, in a paper presented at a conference on music and science held in 2014 at the Moscow State Tchaikovsky Conservatory, György Darvas remarked that the asymmetric nature of the human brain is a consequence of a series of morphological violations that have been taking place at molecular level since primordial times (Darvas 2007). Another noteworthy medieval concept, which still holds its currency today, is the notion that different musical traits can influence our body in specific ways, in particular our brain. Burgeoning research into musical neuroscience has been providing increasingly strong evidence that music does indeed affect brain functioning and plasticity. For instance, it is possible to produce neural correlates of emotions with music (Schmidt

3 MUSIC NEUROTECHNOLOGY 3 and Trainor 2001; Daly et al. 2014) and it has been demonstrated that music can be used as a complementary non-pharmacological treatment for symptoms of dementia (Svansdottir and Snaedal 2006), to cite but two examples. The following section introduces our BCMI project, which is aimed at the development of assistive music technology to enable people with severe physical disabilities to make music using brain signals. In addition to building the technology, we are particularly interested in developing approaches to compose music with it. Next, we introduce Corpus Callossum, a piece of music for a chamber orchestra, which was composed using fmri brain scans 2. Here we explain the brain-inspired methods that we developed for musical composition. The paper ends with a brief concluding commentary. 2 BCMI: BRAIN-COMPUTER MUSIC INTERFACING Brain-computer interfacing technology, or BCI, allows a person to control devices by means of commands expressed by brain signals, which are detected by means of brain monitoring technology (Dornhege et al. 2007). We are interested in developing braincomputer music interfacing technology (Figure 1), or BCMI, aimed at people with special needs and music therapy, in particular for people with severe physical disability who have relatively preserved cognitive functions. Severe brain injury, spinal cord injury and locked-in syndrome result in weak, minimal or no active movement, which therefore prevent the use of gesture-based musical instruments. These patient groups are currently either excluded from music recreation and therapy, or are left to engage in a considerably less active manner through listening (or receptive) methods only (Miranda et al. 2011). 2.1 Approaches to BCMI Currently, the most viable and practical method of detecting brain signals for BCMI is through the electroencephalogram, abbreviated as EEG 3, recorded with electrodes placed on the scalp. The EEG expresses the overall electrical activity of millions of neurones, but it is a difficult signal to handle because it is extremely faint: it is filtered by the membranes that separate the cortex from the skull (meninges), the skull itself and 2 fmri stands for functional magnetic resonance imaging. It is a technique for measuring brain activity, which works by detecting the changes in blood oxygenation and flow that occur in response to neural activity. It works upon the principle that when a brain area is more active it consumes more oxygen and to meet this increased demand blood flow increases to the active area. fmri can be used to show which parts of the brain are involved in a particular mental process. 3 The EEG is a measurement of brainwaves detected using electrodes placed on the scalp. It is measured as the voltage difference between two or more electrodes on the surface of the scalp, one of which is taken as a reference. Other methods for measuring brain activity include MEG (magnetoencephalography), PET (positon emission tomography) and fmri (functional magnetic resonance imaging), but they are not practical for BCI.

4 4 E. R. MIRANDA the scalp. This signal needs to be amplified significantly and analyzed in order to be used in a BCI. In BCI research, it is often assumed that: (a) there is information in the EEG that corresponds to different cognitive tasks, or at least a function of some sort, (b) this information can be detected and (c) users can be trained to produce EEG with such information voluntarily (Miranda 2010). In general, power spectrum analysis is the most commonly used method to analyze the EEG; please refer to (Miranda and Castet 2014) for an overview of EEG analysis methods. In simple terms, power spectrum analysis (Fast Fourier Transform, or FFT) breaks the EEG signal into different frequency bands and reveals the distribution of power between them. This is useful because it is believed that specific distributions of power in the spectrum of the EEG can encode different cognitive behaviors. Figure 1: A BCMI system extracts information from the user s EEG to control musical systems. In this photo, a person is playing a Disklavier MIDI piano through a BCMI in our research laboratory. As far as BCI systems are concerned, the most important frequency activity in the EEG spectrum sits below 40Hz. There are five, possibly six, recognized bands of EEG activity below 40Hz, also referred to as EEG rhythms, which are often associated with specific states of mind. For instance, the frequencies falling between 8Hz and 13Hz are referred to as alpha rhythms and are usually associated with a state of relaxed wakefulness. The exact boundaries of these bands are not so clearly defined and the meaning of these associations can be contentious. In practice, however, the exact meaning of EEG rhythms is not so crucial for a BCI system. Rather, what is crucial is to be able to establish whether or not users can produce power within distinct frequency bands voluntarily. For instance, we have used alpha rhythms to implement an early proof-of-concept BCMI system, which enabled a person to switch between two types of generative algorithms to produce piano music (Figure 1) in the style of Robert Schumann (when alpha rhythms were detected in the EEG) and Ludwig van Beethoven (when alpha rhythms were not detected) (Miranda 2006).

5 MUSIC NEUROTECHNOLOGY 5 Broadly speaking, there are two approaches to steering the EEG for a BCI: conscious effort and operant conditioning. Conscious effort induces changes in the EEG by engaging in specific cognitive tasks designed to produce specific EEG activity (Miranda et al. 2004; Curran and Stokes 2003). The cognitive task that is most often used in this case is motor imagery because it is possible to detect changes in the EEG of a subject imagining the movement of a limb, such as a hand (Dornhege et al. 2007). Operant conditioning involves the presentation of a task in conjunction with some form of feedback, which allows the user to develop a somewhat unconscious control of the EEG (Kaplan et al. 2005). In between the two aforementioned approaches sits a paradigm referred to as evoked potentials, which is the paradigm that we have adopted for the system that will be introduced below. Evoked potentials (EP) are spikes that appear in the EEG in response to external stimuli. EP can be evoked from auditory, visual or tactile stimuli producing auditory (AEP), visual (VEP) and somatosensory (SSEP) evoked potentials, respectively. It is very difficult to detect the electrophysiological response to a single event in an ongoing EEG stream. However, if the person is subjected to repeated stimulation at short intervals (e.g., 9 repetitions per second, or 9Hz) then the brain s response to each subsequent stimulus is evoked before the response to the prior stimulus has decayed. This prevents the signal to return to a baseline state. Rather, it produces a steady-state response, which can be detected with no major difficulties. Steady-state visual evoked potential (SSVEP) is a robust paradigm for a BCI, provided the user is not severely visually impaired. Typically, visual targets are presented to a user on a computer monitor representing tasks to be performed. These could be spelling words from an alphabet or selecting directions for a wheelchair to move, and so on. Each target is encoded by a flashing visual pattern reversing at a unique frequency; e.g., 11Hz. In order to select a target, the user must simply direct their gaze at the flashing pattern corresponding to the action he or she would like to perform. As the user s spotlight of attention falls over a particular target, the frequency of the unique pattern reversal rate can be accurately detected in his or her EEG through spectral analysis. It is possible to classify not only a user s choice of target, but also the extent to which he or she is attending it. This gives scope for SSVEP-based BCIs where each target is not a simple binary switch but can represent an array of options depending on the user s level of attention. Effectively, each target of a BCI-based SSVEP system can be implemented as a switch with a potentiometer. This immediately suggests a number of musical applications. 2.2 ICCMR s first SSVEP-based BCMI system In 2011 we completed the implementation of our first SSVEP-based BCMI system, which we tested with a patient with locked-in syndrome at the Royal Hospital for Neuro-disability, in London.

6 6 E. R. MIRANDA Figure 2: A patient with locked-in syndrome testing the BCMI. The system comprised four targets, as shown on the computer screen in front of the patient in Figure 2. Each target image represents a different musical instrument and a sequence of notes (Figure 3). Each image flashes reversing its colour (in this case the colour is red) at different frequencies: 7Hz, 9Hz, 11Hz and 15Hz, respectively. Thus, for instance if the person gazes at the image flashing at 15Hz, then the system will activate the xylophone instrument and will produce a melody using the sequence of six notes that was associated with this target; these notes are set beforehand, and the number of notes can be other than six. The more the person attends to this icon, the more prominent is the magnitude of his or her brain s SSVEP response to this stimulus, and vice-versa. This produces a varying control signal, which is used to produce the melody. Also, it provides a visual feedback to the user; the size of the icon increases or decreases as a function of this control signal. The melody is generated as follows: the sequence of six notes is stored in an array, whose index varies from one to six. The amplitude of the SSVEP signal is normalized so that it can be used as an index that slides up and down through the array. As the signal varies, the corresponding index triggers the respective musical notes stored in the array (Figure 4). The system requires just three electrodes on the scalp of the user: a pair placed on the region of the visual cortex and a ground electrode placed on the front head. We use active electrodes and their impedances are kept below 5Ω by parting hair at openings in electrode cap and using conductive gel. Filters are used to reduce interference of AC mains noise and artifacts such those generated by blinking eyes or moving facial muscles. SSVEP data was then filtered via parallel filters to extract band power across the frequencies correlating to the flashing stimuli.

7 MUSIC NEUROTECHNOLOGY 7 Figure 3: Each target image is associated with a musical instrument and a sequence of notes. Figure 4: Notes are selected according to the level of the SSVEP signal. The patient took approximately 15 minutes to learn how to use the system and she was able to quickly learn how to make melodies by increasing and decreasing the level of her SSVEP signal. Suggestions and criticism from the staff of the hospital and the patient with respect to improvements and potential further developments were collected (Miranda et al. 2011). Two important challenges emerged from this exercise:

8 8 E. R. MIRANDA (a) The system produced synthesized sounds and the music sounded mechanical; it lacked expressivity. Therefore, the music should preferably be played with real acoustic musical instruments. (b) Our system enabled a one-to-one interaction with a musical synthesiser. However, it was immediately apparent that it would be desirable to design a system that would promote interaction amongst the participants. Therefore, a BCMI should to enable a group of patients to make music together. 2.3 ICCMR s second SSVEP-based BCMI and Activating Memory In order to address the abovementioned challenges, we produced a new version of the BCMI system, and a bespoke composition entitled Activating Memory. Activating Memory is piece for a string quartet and a BCMI quartet. Each member of the BCMI quartet is furnished with our new SSVEP-based BCMI system, which enables him or her to generate a musical score in real-time. That is, each member of the BCMI quartet generates a part for the string quartet, which is displayed on a computer screen for the respective performer to sight-read during a performance. The new BCMI system works similarly to the one described in section 2.2, with the fundamental difference that the visual targets are associated with short musical phrases. Moreover, instead of flashing images on a computer monitor, we designed a device with flashing LEDs and LCD screens, which display what the LEDs represent (Figure 5). The LCD provides an efficient way to change the set of options available for selection. Also, this device increases the SSVEP response to the stimuli because it enables us to produce more precise flashing rates than the ones that we were able to produce using standard computer monitors. Activating Memory was composed as a musical game involving four players. The composition method is inspired by Arca Musurgica, an extraordinary musical device built in 1650, in Rome, by Jesuit Father Athanasius Kircher, and described in his book Musurgia Universalis (Kircher 1650). Kircher s device consisted of a box holding a number of wooden slats. Each of them contained a set of numbers, corresponding to sets of notes and rhythmic patterns. These materials could be combined in a number of ways to form compositions. In the 18th century, a number of musical dice games inspired by Arca Musurgica, appeared in Europe, including Wolfgang Amadeus Mozart s famous Musikalisches Würfelspiel. For this piece, Mozart wrote a series of short phrases, which could be selected randomly, based on the rolls of a dice, and sequenced to form minuets and trios (Mozart 2003).

9 MUSIC NEUROTECHNOLOGY 9 Figure 5: Photo of our new SSVEP stimuli device. In Activating Memory, different choices of musical riff are displayed on the LCD screens instead of numbers 1, 2, 3 and 4. Figure 6: An example of two sets of four musical riffs on offer for two subsequent sections the violoncello part. Activating Memory is composed on the fly by combining sets of programmed musical phrases, or riffs. For each section, the system provides four choices of musical phrases, or riffs, for each part of the string quartet, which are selected by the BCMI quartet (Figure 6). The selected riffs for each instrument are relayed to the computer monitors facing the string quartet for sight-reading. While the string quartet is playing the riffs for a section, the system provides the BCMI quartet with another set of choices for the next section. Once the current section has been played, the chosen new riffs for each instrument are subsequently relayed to the musicians, and so on. In order to give enough time for the BCMI quartet to make choices, the musicians repeat the respective

10 10 E. R. MIRANDA riffs four times. The system follows an internal metronome, which guarantees synchronization of all parts. Activating Memory has been publicly performed on a number of occasions but not with disabled patients yet. This allowed us to make final adjustments to the system and music. As we write this paper, we are in the process of organizing a trial at the hospital with four patients in the summer of In the meantime, we already started building a new version of the system, which will include usage of the amplitude of the SSVEP signal and a new EEG measurement that is able to reveal the affective (or emotional ) state of the BCMI user (Eaton et al. 2014). 2.4 Symmetry of brain activation for a BCMI with emotions The circumplex model of affect introduced in (Russell 1980) provides a way of parameterizing emotional responses to musical stimuli in two dimensions: valence (or positivity) and arousal (or energy of activation). Figure 7: Quadrants with 12 discrete affective descriptors from the Russell s circumplex model (Eaton et al. 2014). We re-organized the circumplex model to create 12 affective categories, which can be indexed via Cartesian co-ordinates (Figure 7). Thus, an emotional trajectory moving from pleased, via happy, to excited, can be represented by a vector which

11 MUSIC NEUROTECHNOLOGY 11 gradually increases in arousal whilst maintaining positive valence: {(v2, a4), (v2, a5), (v2, a6)}. In order to measure EEG correlates of valance and arousal we are placing electrodes on the front of the scalp, over the prefrontal cortex, which is a region of the brain believed to handle emotion (Ramirez and Vamvakousis 2012). Strong EEG activity in the spectral region between 8Hz and 13Hz is known to indicate a relaxed state of mind, and this combined with increased activity in spectral region between 13Hz and 30Hz indicate arousal; i.e., alertness in mental activity. The symmetry of activation levels across the left and right hemispheres indicates a difference between a motivated approach and a more negative, withdrawal type of mental state, which is directly related to valence. We are currently designing and conducting a number of tests with transformative music algorithms to generate music representing the affective descriptors shown in Figure 7 (Williams at al. 2014). We envisage a scenario whereby those transformative algorithms would alter the SSVEP-selected riffs in order to convey a specific mood or mood trajectory. 3 MUSICAL CREATIVITY WITH NEUROSCIENCE Musicians have an extraordinary opportunity today to develop new approaches to composition that would have been unthinkable a few years ago. The second half of this paper will examine the impact of Music Neurotechnology on musical creativity. Corpus Callosum is piece for chamber orchestra composed by means of a new approach to musical composition combining sophisticated brain imaging technology, musical Artificial Intelligence and ideas from Symmetry theory. In Chapter 12 of the book Guide to Brain-Computer Music Interfacing (Miranda and Castet 2014) we introduced this new approach in the context of a composition for orchestra entitled Symphony of Minds Listening, which is a precursor of Corpus Callosum. Symphony of Minds Listening is an experimental symphonic piece in three movements based on the fmri brain scans taken from three different persons while they listened to the second movement of Ludwig van Beethoven s Seventh Symphony: a ballerina, a philosopher and a composer (the author of this paper). In simple terms, we deconstructed the Beethoven movement to its essential elements and stored them with information representing their structural features. Then, we reassembled these elements into a new composition, using the same instrumentation as for Beethoven s 7th symphony, but with a twist: the fmri information influenced the process of reassembling the music. Please refer to (Miranda et al. 2014) for more details; below we present only a brief overview of this method.

12 12 E. R. MIRANDA Figure 8: A typical representation of an fmri snapshot, showing 8 transversal slices of the brain. The fmri technique measures brain activity by detecting associated changes in blood flow. The measurements can be represented graphically by colour-coding the strength of activation across the brain. Figure 8 shows a typical representation of an fmri scan of a person listening to music, displaying the activity of the brain at a specific window of time. In this case, each time window lasts for two seconds. The figure shows 8 planar surfaces, or slices, from the top to the bottom of the brain, and the respective activity detected in these areas. Figure 9 shows an example of a 3D rendition of such an fmri scan, devised by Dan Lloyd, of Trinity College in Hartford, USA: it displays different areas of the brain, represented by different colours (or, shades of grey), responding in a coordinate manner to the music. Each scanning session generated sets of fmri data, each of which we associated to a measure of the second movement of Beethoven s 7 th symphony. This is shown schematically in Figure 10. Figure 9: An artistic 3D rendering of an fmri scan.

13 MUSIC NEUROTECHNOLOGY 13 The score of Beethoven s movement was deconstructed by means of a piece of software of our own design, which extracted statistical information about the structure of the music and used this information to reconstruct it. The process of reconstruction, however, is influenced by the fmri data. Effectively, during the reconstruction process the fmri data altered the original music. Not surprisingly, the scanned fmri data differed amongst the three listeners. Therefore, brain activity from three different minds yielded three different movements for the resulting composition, each of which bearing varied degrees of resemblance to the original symphony. Figure 10: The result of a scanning section is a set of fmri data for each measure of Beethoven s piece. (Note: this is only a schematic representation; the brain imaging does not correspond to the actual music shown.) 3.1 Corpus Callosum, creativity and brain asymmetry Corpus Callosum, for an ensemble of 25 performers, revisits and further develops Symphony of Minds Listening s compositional methods. Our ambition to convey literal musical representations of the data, while remaining faithful to the original form of Beethoven s movement, heavily constrained our musical imagination during the composition of Symphony of Minds Listening. In Corpus Callosum, however, we allowed more freedom to the way in which we handled the materials produced by the computer. And this time we worked with fmri data from this author s own brain only. The title of the composition refers to the part of the brain that connects its left and right hemispheres, and facilitates communication between them: the corpus callosum. The human brain is divided into two asymmetric hemispheres. The left hemisphere is

14 14 E. R. MIRANDA largely engaged in processing the details of things. It is often associated with a more objective, or scientific, knowledge of the world. Conversely, the right hemisphere is largely engaged in taking a more holistic view of things and is often associated with a more subjective, or poetic, interpretation of the world. In a normal brain, the two hemispheres work together, are highly interconnected, and interact through the corpus callosum. Philosopher Friedrich Nietzsche suggested that great artistic creations could only result from the articulation of a mythological dichotomy referred to as the Apollonian and Dionysian. In ancient Greek mythology, Apollo is the god of the sun and is associated with rational and logical thinking, self-control and order. Conversely, Dionysus is the god of wine and is associated with irrationalism, intuition, passion and anarchy. These two gods represent two conflicting creative drives, constantly stimulating, provoking one another. The notion that the Apollonian and the Dionysian tend to counter each other reminds me of the way in which the brain functions at all levels. Inhibitory processes pervade the functioning of our brain, from the microscopic level of neurones communicating with one another, to the macroscopic level of interaction between larger networks of millions of neurones. The book Thinking Music (Miranda 2014) offers a deeper discussion on the relationship between brain asymmetry and the cognitive pushand-pull behind musical creativity. During the scanning session to collect the data for Symphony of Minds Listening, this author remembers that sometimes he lost concentration on the music and his mind wondered off. We reckon that this is a typical manifestation of brain asymmetry at work: while one side of his brain was striving to pay attention to musical detail, the other was making mental associations, producing imageries, eliciting feelings, and so on. The ensemble for Corpus Callosum is divided into two groups: one to be placed on the left side of the stage and the other on the right side. The group on the left side represents the left hemisphere of the brain, whereas the group on the right side represents the right hemisphere. The composition develops as an interaction between these two realms (Figure 11). The instruments on the right hand side 4 of the stage play segments that were composed emphasizing orchestration. These passages do not have rhythm or melody. Rather, the instruments play clusters of sustained notes. Here the focus is on timbre. Conversely, the instruments on the left hand side 5 play modifications of passages pinched from Beethoven s score. These are noticeably rhythmic and melodic segments; timbre is deemed secondary here. 4 2 violins, 1 viola, 1 violoncello, 2 flutes, 2 oboes, 2 clarinets, 1 bass clarinet, 2 bassoons, 2 French horns, 2 trumpets, 1 trombone, 1 bass trombone and percussion (2 players). 5 2 violins, 1 viola, 1 violoncello and 1 marimba (or piano).

15 MUSIC NEUROTECHNOLOGY 15 The piece was composed with the aid of pieces of software that generated the orchestrations and made the musical modifications based the composer s fmri data. Before we explain how these pieces of software work, let us briefly explain how the fmri data were dealt with. The time resolution of the Siemens Allegra 3T scanner that we used to collect the brain data is 2 seconds. That is, it took 2 seconds to take a snapshot comprising 36 image slices of the brain, like the ones shown in Figure 8. Each slice comprised 64 x 64 picture elements, known as voxels, or volume pixels. Thus, each snapshot comprised approximated 150,000 continuously varying voxels. Figure 11: Excerpt from the score of Corpus Callosum, showing only the string quartet at the top and the woodwinds. The first 3 measures hold music representing the left side of the hemisphere whereas the last 4 measures hold music representing the right side.

16 16 E. R. MIRANDA The subjects were scanned listening the second movement of Beethoven's Seventh Symphony twice. The scanning began with 30 seconds without music, then 460 seconds of Beethoven music, then 18 seconds without music, and finally more 460 seconds of Beethoven again. Thus each run generated 484 snapshots. The raw fmri data were first pre-processed following standard procedures for functional neuroimaging using Statistical Parametric Mapping software (Ashburner et al. 2013). Each of the 484 snapshots produced 150,000 voxels, which are very complex for direct analysis. Instead, the image series were further processed with Independent Component Analysis, or ICA (Stone 2004). Informally, ICA separates ensembles of voxels that oscillate in unison. These are unified as supervoxels representing temporally coherent networks of brain activity. The coloured patches on the 3D renditions shown in Figures 9 and 10 correspond to ICA components. A total of ICA 25 components were calculated from the fmri data. In order to rank these components in order of musical significance, the activity of each component during the first pass through the Beethoven listening was compared to that same component during the second pass. If these two segments of a component time series were correlated (with at least p<.05), we hypothesized that the activity might be musically driven, since the stimulus, that is, the music, would be identical at the corresponding time points in the two passes through the music. The order of strength of the 25 ICA components is as follows: 25, 15, 14, 8, 5, 10, 11, 18, 6, 2, 4, 1, 17, 16, 13, 20, 21, 3, 22, 24, 12, 7, 9, 23 and 19. For practical reasons, the actual ICA values were normalized to range from 0 to 9. As a last step, the varying components were resampled to match the timing of the Beethoven score measure by measure. Thus, each time point was indexed to a measure of the Beethoven score. The movement comprises 278 measures therefore each ICA component comprises a time series of 278 values, ranging from 0 (meaning lowest fmri intensity) to 9 (highest fmri intensity). As an example, Table 1 shows the values of the first 5 strongest ICA components (that is, 25, 15, 14, 8 and 5, with p<.002) for the first 10 measures of Beethoven s music, yielded by the fmri of this author. Beethoven Measure ICA 25 ICA 15 ICA 14 ICA ICA 5 Table 1: The values of the strongest 5 ICA components for the first 10 measures of Beethoven s music.

17 MUSIC NEUROTECHNOLOGY Right side: generative orchestrations The materials to compose for the group of instruments representing the right side of the brain were created with a piece of software that generates orchestrations. We modified of a program called ATO-MS (Maresz 2013) to enable it to take into account brain data to generate the orchestrations 6. The process of generating the orchestrations for Corpus Callosum is illustrated in Figure 12. The system takes a given audio file, which is a section from the recording of Beethoven s symphony, and analyses its spectrum using Fast Fourier Transform (FFT) with a window lasting for two seconds. This roughly corresponds to 1 measure of music. This analysis produces a set of 25 frequencies values in Hz, which are the frequencies of the first 25 most prominent partials of the spectrum for every 2 seconds of audio. Thus, for instance, if the audio lasts for 8 seconds, then the analysis produces 4 sets of 25 frequency values each. Figure 12: The process of generating orchestrations. 6 ATO-MS was originally created at IRCAM, or Institut de Recherche et Coordination Acoustique/Musique. This is a French institute for science, music and sound created by composer Pierre Boulez in 1970s, which also promotes the creation of contemporary classical music. It is affiliated to the Centre Georges Pompidou in Paris.

18 18 E. R. MIRANDA The partials of the spectrum of a sound are normally described by means of frequency values and their respective amplitudes (i.e., power). However, we discarded the amplitudes produced by the FFT analysis. Instead we replaced the amplitudes to the partials by the intensity of the 25 ICA components associated to the respective measures of Beethoven s symphony. This results in what is referred to in Figure 12 as the fmri modulated spectrum. The audio database in Figure 12 contains spectral analysis of recordings from all instruments of a symphonic orchestra. It holds analyses of all musical notes that can be produced by those instruments, including different levels of loudness, kinds of articulations and playing techniques. Figure 13: The audio recording of bars lasts for 8 seconds. Given an instrumentation specification, that is the list of instruments that one wants the system to produce orchestrations with, the system searches the database for combinations of instruments whose blended spectrum best approximate the fmrimodulated spectrum. The system produces a number of suggestions for the composer to work with. As an example, let us examine how the 4 th measure of the excerpt shown in Figure 11 was composed. The audio segment (Figure 13) that was used as a target for the orchestration of the last 4 bars of the score shown in Figure 11 corresponds to bars of the 2 nd movement of Beethoven s 7 th symphony. The FFT analysis of the first 2 seconds of this sound produced 25 frequencies, as shown in Table 2. The ICA coefficients for the fmri scan taken at the moment the composer was listening to bar 139 of the symphony are also shown in Table 2. These are the ICA coefficients that replaced the original amplitudes calculated by the FFT analysis, producing the fmrimodulated spectrum as a result. In this case the instrument specification consisted of 2 flutes, 2 oboes, 2 clarinets, 1 bass clarinet and 2 bassoons. Amongst the various suggestions generated by the system, the composer selected the chord shown in the 4 th measure of score shown in Figure 11. Note that one oboe was not used. Also note that individual musical notes of the chord

19 MUSIC NEUROTECHNOLOGY 19 have different dynamics. It is important that the players observe this during the performance in order to produce the desired spectral behavior 7. Partial Frequency in Hz Amplitude (0 9) (ICA 25) (ICA 15) (ICA 14) (ICA 8) (ICA 5) (ICA 10) (ICA 11) (ICA 18) (ICA 6) (ICA 2) (ICA 4) (ICA 1) (ICA 17) (ICA 16) (ICA 13) (ICA 20) (ICA 21) (ICA 3) (ICA 22) (ICA 24) (ICA 12) (ICA 7) (ICA 9) (ICA 23) (ICA 19) Table 2: The fmri-modulated spectrum corresponding to bar 139 of the 2nd movement of Beethoven s 7 th symphony Left side: musical transformations The materials to compose for the group of instruments representing the left side of the brain were created using a number of transformation algorithms. These are algorithms that modify a given musical sequence. The amount of modification is scaled according to an index, referred to as the fmri_index. Effectively, this is the value of the ICA 7 In fact, the amplitudes of the fmri-modulated spectrum are not static. They vary within the 2 seconds long window.

20 20 E. R. MIRANDA analysis extrapolated from the fmri scans, with values between 0 and 9. In order to use the fmri_index as a control signal (CS) for the transformation algorithms, the data is scaled to a range between 0.1 and 1.0. The system applies the following simple scaling process to the value of the fmri_index: CS = {(fmri_index + 1) * 0.1}. A difference value d between the input and the transformed music is also calculated. This difference is then multiplied by CS to give a final scaled modifier value: SMV. The value of SMV is added to the input signal to directly transform the output. This gives a degree of fmri-controlled variability in each transformation: a high fmri_index value will result in major transformations of the music, whereas a low fmri_index value will result in minor transformations. Two of the transformation algorithms are explained below with examples illustrating the effect of varying the value of the fmri_index: pitch inversion and pitch scrambling Pitch inversion algorithm Given an input musical sequence, the pitch inversion algorithm creates a new sequence, which is the input sequence turned upside-down. For instance, a sequence rising in pitch would descend in pitch after being passed through this transformation. In order to illustrate this, let us consider the example in Figure 14. The notes {B4, B4, D5, C5, B4} are converted into MIDI values 8 as follows {71, 71, 74, 72, 71}. Figure 14: An example of a short musical sequence. Pitch inversion is achieved simply by subtracting the current MIDI pitch value from 128 (which is the highest possible MIDI note). For instance, the transformed pitch values for our example created using this technique would be as follows: ( = 57), ( = 57), ( = 54), ( = 56) and ( = 57). The resulting MIDI values are 57, 57, 54, 56 and 57, yielding the following inverted pitch sequence {A3, A3, F#3, G#3, A3} (Figure 15). Figure 15: Newly inverted sequence, after transformation of measure in Figure MIDI is a technical standard for encoding music that allows a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another. MIDI uses a range of 128 pitch values, from 1 to 128.

21 MUSIC NEUROTECHNOLOGY 21 The example above assumed a maximal fmri index value of 9, which once scaled to create a CS gives 1.0. However, varied degrees of transformations are possible by scaling the amount of transformation in function of the fmri_index value. The difference between the input and the transformed pitches is multiplied by CS, before being summed with the input to create the final transformed output value, as follows: New_pitch = {Input_pitch + ((Input_pitch transf_pitch) * [(fmri_index + 1) * 0.1])} Let us examine what happens if we assume an fmri_index equal to 5, which yields a CS equal to 0.6. In this case, we should expect an output approximately half way between the original pitch and the inversion; in other words, an almost neutral set of intervals. First, the difference d between the maximal inversion and the input signal for each of the MIDI values needs to be calculated as follows: d = {(57 71), (57 71), (54 74), (56 72), (57 71)} d = { 14, 14, 20, 16, 14} Then, the scaled modifier values are calculated by multiplying the respective d values by the value of CS, which is equal to 0.6, as follows: SMV= {( 14 * 0.6), ( 14* 0.6), ( 20 * 0.6), ( 16 * 0.6), ( 14 * 0.6)} SMV = { 8.4, 8.4, 12, 9.6, 8.4} Finally, the SMV values are summed with the original input to give a transformed set of output values: New_pitches = {(71 8.4), (71 8.4), (74 12), (72 9.6), (71 8.4)} New_pitches = {62.6, 62.6, 62, 62, 62.6} Pitch values are rounded up to the nearest whole number in order to match the MIDI standard, giving a transformed set of pitch values equal to {63, 63, 62, 62, 63}, which is rendered as {D#4, D#4, D4, D4, D#4}, as shown in Figure 16. Figure 16: Sequence after inversion with fmri_index = 5, giving a nearly neutral set of pitch intervals Pitch-scrambling algorithm In simple terms, the pitch-scrambling algorithm orders the pitch values of the input signal into a numerical list, which is then re-ordered randomly. This provides a stochastic component to the transformation algorithm. Using the same measure as for

22 22 E. R. MIRANDA the previous example (Figure 14) as a starting point, let us examine the result of applying this transformation, as follows: Input pitches: {71, 71, 74, 72, 71} Order pitches in ascending order: {71, 71, 71, 72, 74} Scramble the order of pitches randomly: {74, 72, 71, 71, 71} Output pitches: {74, 72, 71, 71, 71} Figure 17: The result from applying the pitch-scrambling algorithm four times on the same input. In this case, the output would be rendered as {D5, C5, B4, B4, B4}. Re-running the transformation, a further three times would give further variants, for example: {72, 74, 71, 71, 71}, {71, 74, 72, 71, 71} and {71, 74, 71, 72, 71}, rendered as {C5, D5, B4, B4, B4}, {B4, D5, C5, B4, B4} and {B4, D5, B4, C5, B4}, respectively, as illustrated in Figure 17. As with the pitch inversion algorithm, the value of fmri_index can be used to create a control signal with which the amount of transformation can be varied. In order to illustrate this, let us assume an fmri_index equal to 3. This gives a CS value of 0.4. Considering the same input measure as before (Figure 14) and the transformed values from the first pitch scramble shown in Figure 17, the values of d between the first scramble and the input signal are calculated as follows: d = {(74 71), (72 71), (71 74), (71 72), (71 71)} d = {3, 1, 3, 1, 0} The scaled modifier values are then calculated by multiplying the difference values by CS = 0.4: SMV = {(3 * 0.4), (1 * 0.4), ( 3 * 0.4), ( 1 * 0.4), (0 * 0.4)} SMV = {1.2, 0.4, 1.2, 0.4, 0} Finally, the SMV values are summed with the values of the original input to give a transformed set of output values: New_pitches = {( ), ( ), (74 1.2), (72 0.4), (71 0)} New_pitches = {72.2, 71.4, 72.8, 71.6, 71} As in the previous example, pitch values are rounded up to the nearest whole number, giving a transformed set of pitch values equal to {72, 71, 73, 72, 71}, which is rendered as {C5, B4, C#5, C5, B4}, as shown in Figure 18. Note that the output is significantly closer in overall structure to the unscrambled input than the first scrambled transformation shown in Figure 17; only the first and third notes changed.

23 MUSIC NEUROTECHNOLOGY 23 Figure 18: Transformed output created by pitch scrambling algorithm assuming fmri_index = 3. 4 CONCLUDING REMAKS The technology and the compositional method developed for Activating Memory and Corpus Callosum illustrate the interdisciplinary nature of Music Neurotechnology research and its benefits to both, scientific research and musical creativity. Activating Memory is an unprecedented piece of music, which is aimed at being much more than mere entertainment or a commodity for the music industry. Here, eight participants can engage in collaborative music making together, where four of them are not required to move. This forms a suitable creative environment for engaging severely physically disabled patients in music making: they are given an active musical voice to playfully interact between themselves and with the musicians of a string quartet. The first public performance of Activating Memory took place in February 2014 at the Peninsula Arts Contemporary Music Festival, Plymouth, UK. Physically disabled patients were not involved in his first performance. However, we are currently working with colleagues at the Royal Hospital for Neuro-disability in London to trial Activating Memory with severely paralyzed patients, with the objective of creating the Paramusical Ensemble (4 paralyzed musicians and 4 string players) and stage its first public performance in the summer of On the research front, we are currently developing new forms of BCMI control other than the SSVEP-based method. For instance, we are developing ways to detect EEG patterns related to emotional states to control music algorithms, such as the transformational algorithms that we developed for Corpus Callosum. We are in the process of adapting those transformational algorithms to work in real-time, in order to generate scores informed by emotional coefficients. ACKNOWLEDGEMENTS A thank the following post-graduate students, colleagues and collaborators from other institutions for their invaluable contributions to the work presented in this paper: Aurelien Antoine (ICCMR PhD student), Joel Eaton (ICCMR PhD student), Duncan Williams (ICCMR Research Fellow), Anders Vinjas (Notam, Norway), Phillipe Esling (Ircam, France), Zoran Jozipovic (New York University, USA), Dan Lloyd (Trinity College Hartford, USA). REFERENCES Ashburner, J. and The FIL Methods Group at UCL (2013). SMP8 Manual. London: Insitute of Neurology, University College London. Available online at (Assessed on 04 November 2013).

24 24 E. R. MIRANDA Curran, E. A. and Stokes, M. J. (2003). Learning to control brain activity: a review of the production and control of EEG components for driving brain-computer interface (BCI) systems. Brain and Cognition 51(3): Daly, I., Malik, A., Hwang, F., Roesch, E., Weaver, J., Kirke, A., Williams, D., Miranda, E.and Nasuto, S. J. (2014). Neural correlates of emotional responses to music: an EEG study. Neuroscience Letters 753: Darvas, G. (2007). Symmetry. Basel: Birkhäuser Verlag. Dornhege, G., del Millan, J., Hinterberger, T., McFarland, D. and Muller, K-R. (Eds.) (2007). Toward Brain- Computer Interfacing. Cambridge, MA: The MIT Press. Eaton, J., Williams, D., Miranda, E. R. (2014). Affective Jukebox: A Confirmatory Study of EEG Emotional Correlates in Response to Musical Stimuli. Proceedings of Joint ICMC SMC th Sound and Music Conference 14th International Computer Music Conference, University of Athens, Greece. Kaplan, A., Ya Kim, J.J., Jin, K. S., Park, B.W., Byeon. J.G. and Tarasova. S.U. (2005). Unconscious operant conditioning in the paradigm of brain-computer interface based on color perception. International Journal of Neurosciences, 115: Kircher, A. (1650). Musurgia Universalis. Two volumes. Rome: Ex typographia Haeredum F. Corbellitti. James, J. (1993). The Music of the Spheres: Music, Science and the Natural Order of the Universe. London: Abacus. Maresz, Y. (2013). On Computer-Assisted Orchestration. Contemporary Music Review 32(1): Miranda, E. R., Lloyd, D., Josipovich, Z. and Williams, D. (2014). Creative Music Neurotechnology with Symphony of Minds Listening. In E. R. Miranda and J. Castet (Eds.), Guide to Brain-Computer Music Interfacing, pp London: Springer. Miranda, E. R. (2014). Thinking Music. Plymouth: University of Plymouth Press. Miranda, E. R. and Castet, J. (Eds.) (2014). Guide to Brain-Computer Music Interfacing. London: Springer. Miranda, E. R., Magee, W., Wilson, J. J., Eaton, J., Palaniappan, R. (2011). Brain-Computer Music Interfacing (BCMI): From Basic Research to the Real World of Special Needs. Music and Medicine 3(3): Miranda, E. R. (2010). Plymouth brain-computer music interfacing project: from EEG audio mixers to composition informed by cognitive neuroscience. International Journal of Arts and Technology 3(2/3): Miranda, E R. (2006). Brain-Computer music interface for composition and performance. International Journal on Disability and Human Development 5(2): Miranda, E. R., Roberts, S. and Stokes, M. (2004). On Generating EEG for Controlling Musical Systems. Biomedizinische Technik 49(1): Nietzsche, F. (1872). Die Geburt der Tragödie aus dem Geiste der Musik. Leipzig: Verlag von E. W. Fritzsch. Mozart, W. A. (2003). Musikalisches Würfelspiel. Sheet music. Mainz: Schott Music. Ramirez, R. and Vamvakousis, Z. (2012). Detecting Emotion from EEG Signals Using the Emotive Epoc Device. Brain Informatics Lecture Notes in Computer Science, pp London: Springer. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology 39(6): Schmidt, L. A., and Trainor, L. J. (2001). Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion 15: Stone, J. V. (2004). Independent Component Analysis: A Tutorial Introduction. Cambridge, MA: The MIT Press. Svansdottir, H. B. and Snaedal, J. (2006). Music therapy in moderate and severe dementia of Alzheimer s type: a case-control study. International Psychogeriatrics 18(4): Williams, D., Kirke, A., Miranda, E. R., Roesch, E., Daly, I., and Nasuto, S. (2014). Investigating affect in algorithmic composition systems. Psychology of Music (Published online before print August 15, 2014, doi: / )