TOWARDS AN AUDITORY REPRESENTATION OF COMPLEXITY

TOWARDS AN AUDITORY REPRESENTATION OF COMPLEXITY Joachim Goßma ZKM Ceter for Art ad Media Istitute for Music ad Acoustics Lorezstraße 19, D-76135 Karlsruhe, Germay gossma@zkm.de ABSTRACT I applicatios of soificatio, the iformatio iferred by the soificatio strategy applied ofte supersedes the amout of iformatio which ca be retrieved by the ear about the object of soificatio. This paper focuses o the represetatio of complex geometric formatio through soud, drawig o the developmet of a iteractive istallatio soifyig escape time fractals as a example. The terms auditory emergece ad formatio are itroduced ad a attempt is made to iterpret them for music compositio, data display ad iformatio theory. The example applicatio,. Audio Fraktal, is a public istallatio i the permaet exhibitio of the Museum for Media Art at ZKM, Karlsruhe. The desig of the audiovisual display system that allows the shared experiece of iteractive spatial auditory formatio is described. The work was produced by the author at the Istitute for Music ad Acoustics at ZKM, Karlsruhe. 1. INTRODUCTION Ulike the geeratio of soud, which ca be described i parameters such as volume, pitch ad timbre, spectral compositio, dyamic developmet etc., the perceptio of soud seems to defy simple parameterizatio, yieldig a broad badwidth of differet acousto-psychological pheomea: The maifoldess of our spatial auditory world, ivolvig speech, oises, music, etc. Complexity o the other had is a term vaguely based o the athropomorphic cocept of iformatio desity[7]. The defiitio of complexity that we adhere to i this presetatio is the degree by which a structure creates emerget formatio that seems to trasced or is ot deductible from the descriptio of its parts. Famous examples are the emergece of color from the properties of atoms combiig ito molecules, or the emergece of itelliget behaviours from at coloies[14]. Formatio is a perceivable property emergig from a complex process. The process of iterpretig emerget formatios from complex processes by a huma olooker has bee covered i much detail i the iterestig works of Has Dieber[7]. Figure 1: The icoic z 2 Madelbrot set From the polymorphism of our auditory perceptio it ca be hypothesized that a ivestigatio of complexity through soud is a worthwhile edeavour a idea which is explored here employig Escape Time Fractals as a example. ETFs have bee a source of ispiratio for computer geeks ad artists ever sice their mid-bogglig ad colourful visual represetatios were widely published[8][13]. Although a umber of soud pieces ispired by the beautiful images were created, may have argued that there is o successful way to geerate a equivalet to the otios of ifiite complexity ad self similarity for the coscious auditory perceptio of a listeer (ot oly for the structure of the compositio) that these structures provide to the eye: May cases of Madelmusic employ the calculus i a rather uspecific fashio comparable to a strage radom-umber geerator ad carry a lot of dead freight by usig the tempered tuig system of wester music, thereby limitig the auditory outcome to everythig that ca be played o a piao, coverig up may very iterestig ad importat subtleties. 2. CONCEPT 2.1. The data, the soificatio ad the listeer Soificatio attempts to make patters withi datasets readable to the huma ear. Depedig o the applicatio, this ca geerate various relatioships betwee the listeer, the souds ad the coveyed iformatio, resultig i differet solutios for the desig of the soificatio. We ca roughly separate two cases: I the first case, the patters i the data are pre-aalyzed ad categorized, ad are represeted as clearly as possible through soud. The mai focus of creatig such a applicatio is the uambiguous desig of the soud, comparable to the creatio of a road sig or guidace system: The preiterpretatio of the data determies the iformatio cotet of the soificatio. This type of soificatio icludes the readig of parameter values from the soud, if the iterpretatio of the parameters is defied beforehad, such as curret weather coditios. I the secod case, the iformatio cotet of the soified data is at least partially obscure oly a geeral categorizatio of the data might be kow. This use of soificatio to fid ukow structures i the data by ear, is also called exploratory soificatio[4]. I this applicatio, the huma hearig ad its psycho-acoustical ad acousto-psychological properties become a key issue[10]. Exploratory soificatio ca be a trasformatio of data values ito a time series played back to the listeer as PCM-audio (audificatio)[11], but may also employ parameter mappig, or apply models that may ope the soificatio to iteractio[4]. It ICAD05-264

attempts to optimize the emergece of those acoustopsychological pheomea, or auditory formatio, from the data, which trasced the descriptio of the soificatio method itself. This meas that the strategy of soificatio should be as ope for uexpected results as possible as we are ot certai about what we are goig to fid i the first place. The medium (computer geerated soud) should be allowed to create a laguage of its ow formig istead of relyig o metaphors or iferred cultural artifacts, which carry the dager of obscurig the iformatio to be retrieved. what we are lookig for is a auditory laguage comparable to the formal laguages of image for the display of iformatio[9]. 2.2. Auditory formatios a music excursus I the field of soud ad music, comparable abstract forms of expressio developed i the 1940ies: The method of electroacoustic compositio was expadig the role of the composer from writig otes to be played ad iterpreted by istrumetalists, to the creatio of the very soud that is heard, played back through a loudspeaker. Composers were exposed to that freedom whe sigal geerators ad tape recorders as well as other electroacoustic istrumets became available. Pierre Schaeffer, oe of the pioeers i the uses of these ew techologies for compositio, started to work i 1943 o a ew type of music, which was based o souds recorded via microphoe to tape, ad was to be experieced through loudspeakers. I musique cocrete, accordig to Schaeffer, the souds should ufold their ow iheret laguage ad ot refer to the sources they were take from. This mode of listeig to abstracted souds projected through loudspeakers was called acousmatic listeig, from which the term acousmatic music was created. I oppositio to the Frech Musique Cocrete Electroic Music was created i Germay, i which oly souds were admitted that were geerated electroically. After early experimets, composer Karlheiz Stockhause itroduced the possibilities of geeratig souds from the superpositio of sie toes to the studio i Cologe[3]. This idea roots i the Fourier Trasformatio ad the cotemporary idea of decostructig soud ito audio quata[2]. Later, composers such as Yais Xeakis cotiued alog this tedecy to formalize ot oly the musical score but the soud itself. The musical laguage developed by these composers has at first geerated cofusio ad eared disaffirmatio by music listeers i the geeral public, who were uable to create a emotioal or emphatic relatioship with the ew souds they were uacquaited with. Nevertheless it has log sice become a vital part of the soud desig for films ad radio plays ad surrouds us ow o a daily basis. It may also be oted that a emotioal or emphatic relatioship to the souds is also somethig should ot be excluded from the evaluatio of exploratory soificatio, as it represets a high-level acoustopsychological pheomeo. Schaeffer s evisioed abstract laguage of soud is aki to what we would like to call auditory formatio. What a composer of Musique Cocrete tries to brig forward i his compositio is very much related to what we would like to emerge from exploratory soificatio: Perceivable abstract auditory structures. I accordace with Schaeffer, we would like to avoid abusig the structures of traditioal music such as the tempered tuig system or sampled piao souds, uless they emerge from the data at had. What we are workig toward with the cocept of auditory formatio is to bridge the gaps betwee compositio/soud desig, data display ad iformatio theory discipliary crevices that eed to be overcome by a successful soificatio to make audible othig but the data. As a side ote, Schaeffer s postulates ad rules which he formulates i his Lettre a Albert Richard are partially very relevat to the desig ad applicatio of soificatio - A more detailed aalysis of the relatioships betwee soificatio ad the historic developmets of Acousmatic Music is worthwhile, especially cocerig the descriptio ad uses of timbre, as well as the strategies of temporal ad spatial orgaizatio of soud. It eeds to be cotiued elsewhere, as it is beyod the scope of this presetatio. 3. INSTALLATION 3.1. Represetig Escape Time Fractals The istallatio Audio Fraktal is a iteractive audio-visual display of exploratory soificatio. The visitor sees the visual represetatio of a Escape Time Fractal projected oto a pedestal i the ceter of the room, i which a graphics tablet is embedded. Usig a iteractio device (wireless 5-butto mouse), the visitor is able to avigate freely through the two dimesioal structure, zoomig i ad out. The visitor ca activate the soificatio by pressig a butto o the iteractio device. A spatialised soud spectrum fills the exhibitio space that correspods to the curret locatio of the iteractio device withi the visual represetatio. Movig the mouse while keepig the butto pressed results i a spectral ad spatial glissado to the ew locatio of the mouse, allowig the visitor to cotiuously sca across the fractal to ivestigate trasitios. Figure 2: Detail from a ETF geerated from z +1 =exp(z /z o ) with a iteratio path (dowscaled). 3.1.1. Recapturig some maths The formulas geeratig ETFs ca be geeralized to: z + 1 = f ( z, z0) (1) I the case of the stadard Madelbrot set (Fig.1), this becomes 2 z + 1 = z + z0 (2) Poits z 0 i the complex plae for which the orbit z does ot ted towards ifiity are iside the set. If the orbit of z +1 has ICAD05-265

ot trasceded the escape horizo after a maximum umber of iteratios, the iteratio is stopped, ad z 0 is declared to be iside the set. A iteratio path (z 1 z ) is thus geerated from each poit z 0. The umber if iteratios or legth of the path are determied by the first z lyig outside the escape horizo, or with the stop criterio. I visual represetatios of the fractal the color for each graphics pixel it is determied from the properties of the resultig path [8]. I most cases, the color is take from a color gradiet ad deotes the umber of iteratios the calculated path cotais before the escape criteria are met. The choice of the color gradiet is up to the perso ivolved i the rederig of the image. Thus, the visual quata from which the ETFimage emerges are scree pixels mapped to the plae of complex umbers: A comparable quatum for the auditory realm ca be foud i basic frequecy compoets such as trigoometric fuctios. Just like scree pixels ca display ay image, accordig to Fourier s theory ay soud ca be decostructed ito or geerated from sie compoets. The base frequecy f 0 = 40 Hz is chose from the dimesio of the escape horizo: Sice the lowest audible frequecy is betwee 16 ad 20 Hz, a step-legth of 2 is mapped to a frequecy of 20 Hz. The possible iteratio steps that lie beyod the escape horizo are disregarded. They are ot calculated ad their frequecies would accordigly be i the subsoic rage. Each iteratio step is thus represeted by a trigoometric fuctio, together they are assembled ito a spectrum. The time-fuctio P(t) of a iteratio path P would therefore be P () t = < ( p 1) = 0 si a ( f t) f ω (3) p is the umber of steps i the path, a f is a amplitude weightig factor. Accordig to logarithmic processig of frequecies i the ear above 500 Hz [10], the 1/f law is applied for the volume of partials above that frequecy (this is a subjective choice made durig experimetatio ad listeig) 3.1.3. Spatialisatio Soud localizatio works best with souds that feature broad spectra or impulses. Pure sie toes are especially hard to localize ad ted to recombie ito a sigle perceived spectrum whe projected oto the listeer from differet directios. I our applicatio we use these recombiat spectra to evelop the visitor with soud: Each sie toe is spatialised i the directio of the represeted iteratio step, takig the origi of the complex plae as a referece poit. Figure 3: Iteratio paths ad their correspodig spectra. 3.1.2. Makig the iteratio paths readable to the ear usig additive sythesis There are several ways to geerate soud from the iteratio paths. Oe of them is a direct audificatio by creatig a iterative Oscillator, aother is the usage of the properties of the paths to geerate parameters for graular sythesis. These strategies will be ivestigated i future applicatios. For ow, we use the iteratio path to create a spectrum by addig up trigoometric compoets. The distace betwee two subsequet poits i the iteratio path is trasformed ito a frequecy, accordig to the quasi-physical equatio of f = z +1 f0 z, f = 0 40Hz (3) Figure 4: Trasformig a path ito a spatially dispersed spectrum surroudig the listeer. More distict localizatios appear if the path is uequally dispersed aroud the origi of the complex plae or certai frequecy compoets oly appear o oe side. Sice we are usig iteratio paths of 500 steps or more, resultig i broad spectra composed of 500 trigoometric compoets, these pheomea produce quite clear localizatios. 3.1.4. A small audio-guide to good old z 2 I geeral, it ca be stated that the souds geerated from the z 2 Madelbrot (Fig.1) fractal fall ito three mai categories: Noise-like, disharmoic ad chord-like. I the cetral kideyshaped area of the fractal, the compoets form dese clusters soudig like bad-filtered oise. The clusters get lower ad deser toward the borders of the area. Eterig the circular pockets, the oise separates ito bads that arrow dow ito ICAD05-266

discrete pitches: The largest pocket (to the right of the kidey) cotais a diad: A octave. The two smaller pockets o top ad bottom of the structure cotai a triad. Goig aroud the brim ito the smaller pockets, each oe splits ito more pitches. The pitches are clearest i the ceter of the pocket, becomig more ad more ustable toward the border. The sub-pockets withi the pockets cotai more ad more complex chords. Exitig the Madelbrot results i the additio of low frequecy compoets. The soud disitegrates ito a seemigly amorphous disharmoic spectrum. Approachig the set from the outside, more ad more compoets are added to the soud as they oes preset become higher ad higher (comparable to a Shepard toe), but the spectrum stays chaotic ad disharmoic util the set is etered, at which poit the spectrum suddely simplifies ito a chord of discrete pitches. The self-similar sub-sets i the border regio of the Madelbrot feature a idividual chord soud each all of them are differet, yet their soud quality is similar. It may be said that they soud as similar ad yet differet as they look. Yet there is a lot more to discover for the ear i the border areas of the fractal. 3.2. Physical Istallatio The setup chose is a pedestal i the ceter of the room, which is equipped with a graphics tablet, while beig used as a projectio scree at the same time. This allows the visitors to gather aroud the visual display sharig the view ad the soud experiece without blockig each other. The souds are projected ito the room through eight speakers mouted at a height of 2.2 m, surroudig the visitors i a eve circle. For low frequecy projectio, a subwoofer was added. This setup allows a highly defied projectio of a spatial soud scee i combiatio with uhidered access to a visual display, all of which ca be shared by a small group of people gatherig aroud the pedestal. 4. CONCLUSIONS AND FUTURE WORK We thik that this strategy to soify escape time fractals, opeig them to a iteractive exploratio, is quite successful at exposig properties of these structures that are ot visible, displayed much more efficietly i a auditory laguage. Certai questios about how these structures are shaped i the way they are ca be aswered ituitively by iteractig with the souds themselves. The use of additive sythesis limits the expressiveess of the soudig result, but it also eables the istallatio to geerate a seemigly ifiite amout of differet spectra each of which carry discerable idividual characteristics. Yet, the possibilities of lettig data pait themselves oto the cavas of our auditory perceptio are by far ot exhausted with this istallatio: future istallatios will explore the possibilities of graular sythesis ad complex oscillators. Pushig the istallatio further alog the lie betwee a istrumet for soificatio ad a tool for electroacoustic music, metaphors are beig developed icludig multi-idexig (each loudspeaker beig ad idex o the complex umber plae) or the geeratio of algorithmic behaviors for automatic motio of a idex across the complex plae ( ) Figure 5: Schematic of the display setup 5. ACKNOWLEDGEMENTS Special thaks to Peter Weibel for curatig this piece ito the permaet exhibitio of the Media Museum at ZKM, ad to Ludger Brümmer of the Istitute for Music ad Acoustics for supportig its productio. Very special thaks go to the people at the Media Museum who did a woderful job i costructig the pedestal ad the room itself ad were very helpful, competet ad friedly i the face of a eormous workload. 6. REFERENCES [1] C. E. Shao: A mathematical theory of commuicatio. Bell System Techical Joural, vol. 27, pp. 379-423 ad 623-656, July ad October, 1948. [2] D. Gabor: : Acoustical Quata ad the Theory of Hearig. Nature, 159 (4044), p. 591-594, U.S.A. 1947 [3] E. Bozzetti: Karlheiz Stockhause: Elektroische Studie II (1954). Musik ud Bildig, pp. 18-20, Maiz, Germay 1973 [4] T. Herma: Soificatio for Exploratory Data Aalysis. PhD Thesis, Bielefeld Uiversity, Bielefeld, Germay, 2002 [5] T.Herma, J.M.Drees,H.Ritter: Broadcastig Auditory Weather Reports a Pilot Project,. Proceedigs of the 2003 Iteratioal Coferece o Auditory Display, Bosto, MA, USA, 6-9 July 2003 [6] M. Supper: Elektroakustische Musik ud Computermusik, Wisseschaftliche Buchgesellschaft, Darmstadt, Germay, 1997 ICAD05-267

[7] Has Dieber (Editor): Studium geerale zur Komplexität. Geista-Verlag, Berkeley CA, U.S.A., 2001 [8] Heiz-Otto Peitge, Peter H. Richter: The Beauty of Fractals, Spriger Verlag, Berli, Heidelberg, New York, Germay, 1986, ISBN 0-387-15851-0 [9] Edward Tufte: The visual display of quatitative iformatio. Cheshire, CT: Graphics Press, U.S.A., 1983 [10] E. Zwicker ad R. Feldkeller, The Ear as a commuicatio receiver, America Istitute of Physics (Acoustical Society of America), U.S.A.,1999 [11] Dr. F. Dombois, Usig Audificatio i Plaetary Seismology. Proceedigs of the 2001 Iteratioal Coferece o Auditory Display, Expoo, Filad, July 29- August 1, 2001 [12] J. Goßma ad Dr. F. Dombois: The Spatial Soud Lab at Frauhofer IMK. Proceedigs of the 2003 Iteratioal Coferece o Auditory Display, Bosto, MA, USA, 6-9 July 2003 [13] http://spaky.triumf.ca/www/fractit/fractit.html [14] Douglas Hofstadter: Goedel, Escher, Bach. Basic Books, U.S.A., 1979 Figure 6: Iside the Audio Fraktal exhibit ICAD05-268