Visualizing the Semantic Structure in Classical Music Works

Transcription

1 Visualizing the Semantic Structure in Classical Music Works Winnie Wing-Yi Chan Supervised by Dr. Huamin Qu MPhil Thesis Defense Examination Department of Computer Science and Engineering The Hong Kong University of Science and Technology January 13, 2009

2 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion 2

3 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o Background Motivations Contributions 3

4 Background (book covers courtesy to amazon) Classical music is generally less friendly and approachable Realizing structure is difficult for unskilled ears The first theme begins presented in a minor key by the violins. Then part of the first theme is repeated by the violins. The transition begins in the violins; it contains a short motive and many scalewise passages. The second theme is presented by the violins and woodwinds. The second theme is repeated by the woodwinds and violins. The music moves on to some transitional material. A motive from the first theme is played by the clarinet and bassoon. The codetta begins with rapidly moving notes played by the violins. The codetta concludes with a solid chord. The development section begins with two short chords and notes played by the woodwinds. A portion of the first theme is played by the violins three times in sequence, each time one note lower than before. A portion of the first theme is played by the lower strings as the upper strings play a contrasting line of rapidly moving notes. The upper and lower strings trade what they play. The upper and lower strings again trade what they play. The violins continue playing portions of the first theme. The motive from the first theme alternates quietly several times between the violins and the flute. Fragments of the first theme are exchanged more energetically between the upper and lower strings. A short link leads to the recapitulation. The first theme returns played by the violins in the tonic key. The first theme continues. Part of the first theme is repeated by the violins. The transition begins Learning with musical scores and essays are demanding 4

5 Motivations Visualizing the semantic structure of music Impress with its sophistication and beauty Reveal underlying musical structure Assist music learning and teaching Previous approaches Rarely on semantic structure for perceptual understanding Mainly on sonic features or low-level physical properties music visualizers examples 5

6 Contributions We pioneer a visualization solution to convey the semantic structure in classical music General users Understanding the semantic structure Appreciating the sophistication of music Music teachers, music students, and musicians Teaching and learning different styles Gaining new insights into features 6

7 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o o o o Musical structure Abstracted scores Performance expression 3D music Commercial products Computer music 7

8 Musical Structure Visualization: Arc Diagrams [InfoVis 02] Extract repeated patterns from a MIDI file automatically May not be musically meaningful Show repetitions of only one track 8

9 Musical Structure Visualization: Isochords [GI 07] Visualize chord structure and progression with animation Show low-level notes and chords Tonnetz 9

10 Musical Structure Visualization: ImproViz [CHI 05] Shown by a melodic landscape and harmony palette Designed for one composition manually 10

11 Abstracted Scores Graphic Scores [EMS 05] Simplified Scores [CW 02] Visual Music [SIGGRAPH 05 Poster] 11

12 Performance Expression Visualization Understand music performance in cognitive terms Suggest that MIDI parameters do not link to human perception Chernoff Faces [Vis 02] Vertical Bar Display [AVI 04] comp-i [SIGGRAPH 03 Sketches] Music & Emotion [ICME 04] 12

13 3D Music Visualization SIGGRAPH 03 Sketches & Applications ICMC 98 VSMM 01 CG&A 07 13

14 Commercial Products TimeSketch Music Animation Machine Hyperscore 14

15 Visualization in Computer Music Self-similarity grid for waveform [ICMC 01] Harmonic visualization [ICMC 01] Self-organization map for tonal content [CIE 05] Spiral layout for tonal evolution [CIE 05] 15

16 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o Formulation Definitions Uniqueness 16

17 Problem Formulation 1) Macro-relationship Vertical texture among layers (depth) Collaborate Accompany Blend Contrast 3) Macro-micro relationship Interactions between layers and themes A layer plays some themes A theme is played by some layers 2) Micro-relationship Horizontal foundation across time (breadth) Themes: Recurring main melodies Repetitions: Balance in time Variations: Interesting details 17

18 Defining the Data: Macro-Level Layers Musically known as parts or voices Generic view of instrument tracks Determined by musical significance, not physical section Layer Roles Functional role of a layer in the harmony of all at a time Change over time Shared by layers in the same functional group Multiple roles at the same time possible Not specified for transitional materials 18

19 Defining the Data: Micro-Level Themes Main melodies and musical ideas Also known as motifs and subjects Only a few in a composition Repeated to bring unity and contrasted to introduce variety Theme Variations Melodic, rhythmic, instrumental, and transformative means May reoccur at different layers 19

20 Data Uniqueness Different levels of viewpoints Coordinated relationships Macro Interrelated layers with different roles Micro Repeated and varied themes across time Macro-micro Macro layer roles vs. micro theme variations 20

21 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion 21

22 Data Preprocessing No existing algorithm to parse high-level structure Manually retrieve from essays Front-end processing can be automated, but is beyond the scope of visualization Processed Data Beginning and ending beat Instruments involved Layer roles and theme variations Processing Beethoven s Symphony No.5, first movement 2 days without computer assistance At most a few hours with simple user interface 22

23 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o Design criteria Color scheme for layers Theme glyph grammar 23

24 Design Criteria Consistent Avoid confusion Assign basic visual channels carefully Intuitive Be comprehended rapidly without training Introduce some familiar metaphor Effective Gain insights into the structure effectively Aesthetic Reflect both artistry and complexity 24

25 Design Overview theme form transformation Woodwind: Brass: 1 2 A B C augmented diminished inverted Percussion: duration singleton Strings: lengthened shortened echoed transposed diff. key A color scheme for layers More flexible than instrument icons A theme glyph grammar with typographical concepts Conveys semantic meanings of the variations 25

26 Color Scheme for Layers: Music and Color Tone Color Quality of a sound Depending on performance style Psychology Emotions invoked Human perception of sound Physics Mapping between sound and color spectrum Rational but lacks perceptual justification None is relevant to semantic structure 26

27 Color Scheme for Layers: Proposed Solution Woodwind: flute oboe clarinet bassoon Brass: horn trumpet trombone tuba Percussion: triangle snare drum timpani bass drum Strings: violin viola cello double bass Hue: physical color of instrument families Lightness: pitch range Percussions in an outlier color Non-orchestral layers based on users' preferences 27

28 Theme Glyph Grammar: Glyph Candidates No universal design which is the best to represent theme, depending on visualization purposes #b 28

29 Theme Glyph Grammar: Typographical Grammar Rules 1 theme 2 Duplicate a sub-part of the glyph Feasible as number of themes is usually limited type size ascender x-height typography form Change the core x-height shape A B C Categorical attribute Overall appearance is preserved with obvious changes duration lengthened shortened transformation augmented diminished inverted Adjust the ascender or descender part Changes to rhythmic pattern or partial appearance One-dimensional variation Transform the core x-height part A theme is structurally transformed into other shapes Only modify x-height part to remain similar and consistent echoed singleton transposed diff. key Add indicative decorations Special variations Indicator can cope with more information if available 29

30 Theme Glyph Grammar: Typography in Our Framework How to size and modify type glyphs Content clearly defined Visual impact formally applied Context to be delivered maximized Proposed grammar can be generalized to any glyph designs for optimal visual impact and legibility x-height part of a lower case letter that equals height of x x d ascender p H descender capline meanline type size baseline 30

31 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o Layer braid Theme fabric 31

32 Visualization Prototypes Layers Themes Macro-relationships among layers Micro-relationships of theme occurrences Macro-micro relationships between layer roles and theme variations 32

33 Textile Art Metaphor Intuitive and efficient with familiar metaphor Similarities with musical structure Delicate cloth as art piece Structure along x- and y-axis Detailed textile pattern (source) 33

34 Layer Braid: Overview flute oboe clarinet bassoon horn I horn II violin I violin II viola cello double bass dominant accompany contrast collaborate ensemble Threads Respective layer colors Interact and correlate over time 34

35 Layer Braid: Plaiting Process layer roles A B C t time Layer threads follow the musical score order Braid patterns determined by interweaving positions in 2D Layers of the same functional group are plaited together Multiple roles: spread, plait, and rejoin Threads may cross unwanted roles during interpolation May darken segments without specific role (source) 35

36 Theme Fabric: Arranging Glyph Nodes layer 1 layer 2 layer 3 time Overlay theme glyphs on a musical score layout x-position: when the theme is played y-position: which layer plays the theme 36

37 Theme Fabric: Stitching Styles Repetitions of glyphs not obvious without visual clues Stitch glyphs in various textile art styles plain style bundled style collapsed style decreasing levels of detail of micro-relationships 37

38 Theme Fabric: Plain Style warp weft plain weaving Connect same glyphs in orthogonal direction Horizontal curved weft Vertical straight warp Utilize color and lightness of threads 38

39 Theme Fabric: From Plain Style to Bundled Style Replicated horizontal threads Bundle horizontal threads Remove redundant vertical threads Maintain betweenness centrality of glyphs butterfly stitches 39

40 Theme Fabric: Bundled Style Reveal more high-level thematic structure Same glyphs across layers are visually clustered Structure is outlined by thread bundles and knots Thread blending: address overlapping Thread patterns: highlight special glyph groups 40

41 Theme Fabric: From Bundled Style to Collapsed Style Collapse identical glyphs at different layers into one single glyph Cluster threads into arcs Become some Arc Diagrams layout drop-stitch 41

42 Theme Fabric: Collapsed Style for Sole Micro View Specific arc height for every unique glyph Split for former and latter glyph Rainbow arc for layers 42

43 Theme Fabric: Collapsed Style for Macro-Micro View Specific arc height for each layer roles Split for former and latter glyph layer role A layer role B Rainbow arc for layers 43

44 Theme Fabric: Collapsed Style for Macro-Micro View Combine arc height with layer roles in layer braid Duplicate arcs for multiple roles 44

45 Theme Fabric: Summary Plain Most expanded and detailed Bundled Connects same glyphs with thread bundles Collapsed Most abstract with compressed layers Macro-micro with arc heights for layer roles 45

46 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion 46

47 User Interactions Customization Zooming and Panning Selection Filtering Linking and Brushing Abstraction Focus + Context Details on Demand Synchronization with Music Configure layer colors and theme glyph type Less flavored than focus + context Select any visual entities on screen Adjust the brightness or opacity value Brush to explore different modules Link different views by overlaying them Render in different levels of detail Aggregate small changes in data View with bifocal display Show more details when zoomed in Animate with music 47

48 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o Macro-relationship Theme and variations Macro-micro confluence 48

49 Macro-Relationships Mozart's Symphony No. 40, 1 st Mov t flute oboe clarinet bassoon horn I horn II violin I violin II viola cello double bass exposition development recapitulation Development exhibits highly complicated relationships Ensembles are only heard once at the end of exposition 49

50 Macro-Relationships Mozart's Symphony No. 40, 1 st Mov t flute oboe clarinet bassoon horn I horn II violin I violin II viola cello double bass Violins are mostly dominant Bassoon becomes dominant more frequently flute oboe clarinet bassoon horn I horn II violin I violin II viola cello double bass 50

51 Theme and Variations Mozart's Symphony No. 40, 1 st Mov t flute exposition exposition (repeated) development recapitulation oboe clarinet bassoon violin I violin II viola cello double bass Not associated with any theme: horn I horn II Blue threads (first subject) are usually played by strings Vivid variations are seen in development and recapitulation 51

52 Macro-Micro Confluence Beethoven s Symphony No. 5, 1 st Mov t The fate motif never dominates the music The inverted, shortened motif in the lower strings contrasts the music at macro-level 52

53 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o User study User survey 53

54 User Study Accuracy Time (min) Prototypes Mean SD Mean SD Layer Braid 86% 12% Theme Fabric (Bundled) 95% 9% Theme Fabric (Collapsed) 89% 15% Details 10 college students 12 questions 5-min introduction Novice users gain insights promptly without prior training A pictorial guidance for music learning and appreciation 54

55 User Survey 10 Experts 7 music students 2 music educators 1 musician Braid and fabric are excellent metaphor of music Layer braid is useful as the layer role concept is abstract Bundled theme fabric is preferred by advanced users Collapsed style is concise and aesthetic Prototypes cooperate with or complement one another 55

56 Expert Feedback Highlights Educators Reading essays diminishes learning interest Layer information is important Visualizing music without clear form? design highly generalized. Students Layer feature helps differentiate instruments Comparing compositions become easier Musician Audience find classical music more approachable Listeners grow from appreciation to learning Improvements Include more attributes, e.g. dynamics, tempo, tone color Add descriptive texts 56

57 Outline Introduction Related Work The Musical Structural Data Data Preprocessing Design Principles Visualization Prototypes User Interactions Case Studies Evaluation Conclusion o o o o Demo Summary Conclusion Future Work 57

58 Demo 58

59 Summary Innovative visualization solution for semantic structure in music Macro-level layer relationships Micro-level theme variations Macro-micro interactions between layers and themes Design principles Color scheme for layers Glyph grammar founded on typography Visualization prototypes derived from textile art Layer braid Theme fabric (plain, bundled and collapsed style) Representative case studies Encouraging user evaluation 59

60 Conclusion Apply information visualization to other discipline Strong human factors Artistic sensation Introduce art and design analogies to address perceptual issues Typography Textile art metaphor Generalize the techniques for applications with semantic context News visualization Story-telling visualization 60

61 Future Work Visualize more compositions to fully realize its potential Implement a simple user-interface for data input Seek the possibility of using the visualization as teaching tool 61

62 Publications Wing-Yi Chan, Huamin Qu, Wai-Ho Mak. Visualizing the Semantic Structure in Classical Music Works. Accepted to IEEE Transactions on Visualization and Computer Graphics (TVCG) with minor revision. Huamin Qu, Wing-Yi Chan, Anbang Xu, Kai-Lun Chung, Kai-Hon Lau, Ping Guo. Visual Analysis of the Air Pollution Problem in Hong Kong. IEEE Transactions on Visualization and Computer Graphics (TVCG), vol.13, no. 6, pp ,

63 Q & A After silence, that which comes nearest to expressing the inexpressible is music. - Aldous Huxley, Music at Night, 1931 flute oboe clarinet bassoon horn I horn II violin I violin II viola cello double bass flute oboe clarinet bassoon violin I violin II viola cello double bass Thank you! 63