Compilers with EARs. Program Auralisation

Transcription

1 Compilers with EARs Program Auralisation 1

2 2 Or debugging with music Vickers, P. & Alty, J. L. (2005). The Welltempered Compiler: The Aesthetics of Program Auralization. Aesthetic Computing. P. Fishwick MIT Press. Vickers, P. & Alty, J. L. (2003). Siren Songs and Swan Songs: Debugging with Music. Communications of the ACM 46(7): Vickers, P. & Alty, J. L. (2002). When Bugs Sing Interacting with Computers 14(6):

3 3 Whoah! Say what? Musical approach helps programmers catch bugs New Scientist (2359, 7/9/02) id=ns Programming tool makes bugs sing Technology Research News (7/8/02) Page_One_ html

4 4 In the news 13 April 2004 Interview on CBS radio! See acfnews summary and transcript of the Osgood File in which it appeared. Full mp3 of interview available at British Council story Article in Science News Ross, D. Music programmes to find bugs IEE Review, 2003, 19. Henderson, T. Sound of music to bring out the bugs The Journal, Newcastle upon Tyne, 2002, 22. Survey fever BBC Music Magazine, 2002, 4. Zimmermann, R. Schräge Töne im Programm: Die Erfindung eines Doktoranden macht Software- Fehler hörbar SonntagsZeitung, Zurich, 2002, 127.

5 5 Rewind Software visualisation uses external visual representations (EVR) Coordination of multiple EVRs e.g. CRUSADE project at Sussex Program comprehension & debugging The use of external auditory representations (EAR) is called software auralisation Often an auditory icon approach Vickers & Alty applied a structured musical framework

6 6 Program auralisation Mapping of program data and events to sound LogoMedia Sonnet ADSL (auditory domain specification language) LISTEN Jackson & Francioni (1992) parallel programs Brown & Herschberger Largely untested (empirically)

7 7 Why music? Gives multiple audio streams in parallel Structured and organised according to specific rules ( organised sound -- Varese) Time-ordered. Visual techniques are spatial. Spatial sound placement enhances the music Gives sense of continuity/continuous flow How can music s grammars be used to carry real information? Vickers, P. and Alty, J.L., Using Music to Communicate Computing Information. Interacting with Computers, (5): pp

8 8 Musical framework Parallel & structured Grammar allows separate musical components to work with rather than against each other Time-ordered Visual techniques primarily spatial Programs operate in temporal domain Audio well suited to monitoring state changes over time

9 9 caitlin system Computer Audio Interface to Locate Incorrect Nonsense Uses musical motifs to render Pascal language constructs at run time Motif design is hierarchic as is organisation of constructs

10 10 Hierarchic motif design Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design. Interacting with Computers, (5): pp

11 11 Design of motifs Points of interest Construct Entry Evaluation of condition Execution of body Exit from construct Boolean True=major, False=minor Drones

12 12 Selection tunes Metaphorically asking & answering a question. Tune rises on entry and descends on exit

13 13 Iteration tunes Major/Minor chord to show when condition evaluated Heard at start of WHILE loops and at end of REPEAT loops FOR loops increase/decrease the pitch each time round the loop

14 14 Variations All selections use same basic tune, modified for individual constructs All iterations use another basic tune, again modified

15 15 Auditory parentheses As () bracket a thought in a sentence, so the auditory parentheses serve to open and close constructs

16 16 Other statements All other statements are currently represented by a drum beat, one per statement.

17 17 Advantages Over Auditory Icons symbolic mappings = poor system aural ecology Syntactic design expect retention Displays both flow and state An execution-specific program structure diagram

18 18 Demonstration

19 19 IF (true) a := 10 ; IF a = 10 THEN Writeln ( A is 10 ) ;

20 20 IF ELSE (false) a := 1 ; IF a = 10 THEN Writeln ( A is 10 ) ELSE Writeln ( A is not 10 ) ;

21 21 CASE ELSE match a := 3 ; CASE a OF 1 : Writeln ( A is 1 ) ; 2 : Writeln ( A is 2 ) ; 3 : Writeln ( A is 3 ) ; ELSE Writeln ( No match ) ; END ;

22 22 WHILE 2 iters cntr := 1; WHILE cntr <= 2 DO cntr := cntr + 1 ;

23 23 REPEAT 3 iterations cntr := 1 ; REPEAT cntr := cntr + 1 ; UNTIL cntr > 3 ;

24 24 FOR TO 6 iters FOR cntr := 1 TO 6 DO Writeln (cntr) ;

25 25 Sequence & nesting Clue to nesting is in the drone If you hear a continuous note then you are inside a construct. Each construct has a drone. For nested constructs you hear multiple drones simultaneously Drone pitch increases as nesting deepens

26 26 Example program Program Exemplar VAR counter : Integer ; BEGIN counter := 1 ; WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln ( Counter is even ) ; counter := counter + 1 ; END ; END.

27 27 Score

28 28 Score

29 29 Structure Diagram

30 30 Experiment 1 Listening test to assess comprehensibility of motifs 22 subjects, 60 auralisations Recognition rate better than by chance and with very little training Hierarchic organisation allowed class level identification & abstraction Vickers, P. and Alty, J.L., Musical Program Auralisation: A Structured Approach to Motif Design. Interacting with Computers, (5): pp

31 Experiment 2 Bug 31 location Questions Do subjects locate more bugs with the additional auralisation information than without? Does the musical experience of subjects affect their ability to make use of program auralisations? 22 subjects, 8 programs, 4 auralised, 4 normal Given: spec, outline design, input, output, & expected output Then, ten minutes to locate bug with either Above documentation, or Above documentation + auralisation Vickers, P. and Alty, J.L., When Bugs Sing. Interacting with Computers, 2002, 14(6):

32 Experiment 2 Bug 32 location Experimental design (within subjects, repeated measures) Group A had 1, 3, 5, & 7 without auralisation and 2, 4, 6, & 8 with auralisation Group B had 1, 3, 5, & 7 with auralisation and 2, 4, 6, & 8 without

33 Experiment 2 Bug 33 location Observe a significant difference in debugging scores: t(21)=2.10, p<0.05 Bug location time not affected:t(21)=1.98, p=0.06 Auralisations led to increase in subjects perceived workload (measured with NASA TLX): t (21)=2.12, p=0.046 Musical knowledge/experience had no effect: regression model 0.16 <= p <= 1.71

34 Experiment 2 Bug 34 location Best candidates were programs where output contained no clues to bug location with a high complexity, complex boolean expressions

35 35 Limitations No comparison with visual task Cognitive latency of motifs? No longitudinal results Restricted to program flow

36 36 Enter Gamelan EPSRC bid to investigate Auditory External Representations for an OOP language (prob. Java, though C# has an obvious appeal ) (c.f. EPSRC CRUSADE project at COGS in Sussex - du Boulay et al) Comparing EAR with EVR in a variety of program comprehension & debugging scenarios Software stethoscope?

37 37 Interesting issues EAR EVR: how to compare? Scalability Parallelism (multi-threading) Knowing when an identifier goes out of scope Music flow may help here Method calling/called-by paths Control flow & Data flow Data structures

38 38 We can envisage Attaching auralisation tags to data structures with predefined/preassigned mappings for different types of data: an integer would sound one way, an array another allowing data and data structures to be inspected at will, their auralizations not being indirect via program flow (as was the case with Alty s bubblesort auralization where the auralization of the list was done via a mapping of the algorithm flow/ state). Could listen to program flow and at any time ask for a variable/data structure update auralisation. This might require suspension of execution during the update so as not to miss any program events.

39 Passive & active 39 auralisation Two basic types of EAR approaches could be developed for data representation: program flow/real-time in which data are auralised as they are modified during execution (passive auralisation) and asynchronous, where auralisations can be requested at any time during execution by button presses/mouse clicks (active auralisation). There are many other possibilities that can be explored

40 40 Coordination Of particular interest will be to explore how multiple EARs are coordinated by users How EARs can be used in combination with EVRs to provide enhanced multi-modal support of program comprehension and debugging

41 41 The power of multimodality? Visualisation Auralisation

42 42 Sonification aesthetics Chaotic attractor functions Mayer-Kress, G., Bargar, R. & Choi, I. (1994). Musical Structures in Data from Chaotic Attractors. Auditory Display. G. Kramer. Reading, MA: Addison-Wesley. XVIII: Earthquake Marty Quinn s Seismic Sonata Pacific ocean buoys Bob Sturm s Music from the Ocean

43 43 Fusion with real music e.g. Ian Boddy: Continuum Intriguing overlap between sonification/ auralisation and generative music & electroacoustic music

44 44 Indicative publications Vickers, P. & Alty, J. L. (2005). The Well-tempered Compiler: The Aesthetics of Program Auralization. Aesthetic Computing. P. Fishwick MIT Press. Vickers, P. (2004). External Auditory Representations of Programs: Past, Present, and Future - An Aesthetic Perspective, in Proc ICAD The Tenth Meeting of the International Conference on Auditory Display, Sydney, 6-9 July. Vickers, P. & Alty, J. L. (2003). Siren Songs and Swan Songs: Debugging with Music. Communications of the ACM 46(7): Vickers, P. & Alty, J. L. (2002). When Bugs Sing. Interacting with Computers 14(6): Vickers, P. & Alty, J. L. (2002). Musical Program Auralisation: A Structured Approach to Motif Design. Interacting with Computers 14(5): Vickers, P. & Alty, J. L. (2002). Using Music to Communicate Computing Information. Interacting with Computers 14(5):

45 45 caitlin musical grammar Uses motif classes and variations to render Pascal constructs Allows abstraction nesting (tonality & harmony preserved across motifs) Uses grammatical rules (e.g. a tonal music system)

46 caitlin musical grammar 46 cont Renders information not directly communicated by source code or other visualisation techniques e.g. construct persistence (nesting) a new grammar, conveys extra meaning same meaning but from different perspective

47 47 Theoretical framework Musical grammars as communication medium Mapping of syntactic features of Pascal to music Construct hierarchy motif hierarchy Motifs organised around tonal musical framework Use of surface & deep characteristics Meter & rhythm more useful than harmony Percussion adds emphasis Drones maintain context

48 48 Real grammar? Translation or transliteration? Have we simply rendered written Pascal as musical Pascal (e.g. written vs spoken English) Like writing I want to go to bed as Ich will gehen zum Bett (transliteration) Is there a new grammar? Ich will zum Bett gehen (translation)

49 49 More auralisation? Is further auralisation needed? Perturbations in flow pinpoint general region of bug. Non-flow bugs very hard to detect anyway & require variable inspection Inspection difficult to render in audio