NISE - New Interfaces in Sound Education Daniel Hug School of Education, University of Applied Sciences & Arts of Northwestern Switzerland April 24, 2015 «New» Interfaces in Sound and Music Education? New technologies, such as video tracking, sensors and multitouch devices, have become widely available Combined with advanced synthesis and signal processing capabilities Apps offer new and accessible possibilities to use advanced music tools and technologies, even without in-depth musical knowledge Are becoming increasingly popular with educators New technologies afford integration of media & modalities, potentially support IME Adequate Music Education related research efforts still in infancy: Need for a reflective approach to technology: ( ) teachers often fail to consider appropriate pedagogical strategies when using technology with music students (Bauer, 2014, p. xiii) Most existing research and applications focus on Apps and conventional note / melody / harmony / rhythm related approaches An in-depth (design) critique of user interface / interaction paradigms is missing 24.4.2015 2 1
Inspiration: New Interface for Musical Expression (NIME) 30.04.2015 3 Music Education Is Also Sound Education Development of sound music in the 20th century Pierre Schaeffer, Karlheinz Stockhausen Jimi Hendrix «Ecological» approach to meaning in music (Clarke 2005) Consequence: Opportunity (both technological and conceptual!) to work with sound as fundamental material of musical expression understanding and creating sound objects building the instrument (the conditions and rules of sound generation), not just playing it Democratization of sound technology proper Digital musicianship is still mostly about moving notes for existing instruments around on a grid. The tools for modulating sound are quite complex. How can we make them accessible? So.. What about "understanding and creating sound" - interfaces? 30.04.2015 4 2
Towards Pedagogically Motivated Design Research Design Research: dealing with possible futures and wicked problems Donald Schön, the reflective practitioner common reference in DR & PR Artifact and activity centered, informed by activity theory (Kaptelinin & Nardi 2006) Designerly ways of knowing" (Cross 2001), abductive, iterative process (Kolko 2010) Multisensory design (Haverkamp 2012) -> Link to Integrated Music Education Methodical approach Participatory Design (Mareis, Held, Joost, 2013) Experience Mockup (Buchenau & Suri 2000) from pen-on-paper to functional prototypes Generation of artifacts and discourse At the same time: applicable in teaching, educational value! Close integration with workshop and classroom settings Towards an integration of pedagogical and design research (Hug 2007, Hug 2010, Hug & Kemper 2014) Integration with existing model of TPACK (Mishra & Koehler 2006, in Bauer 2014), extending it with the DESIGN dimension. Technological Pedagogical Content Knowledge. Reproduced by permission of the publisher 2013 by tpack.org 30.04.2015 5 Partners & The Four Fields of Action of «NISE» Partners FHNW: School of Education, Music Education Sek I&II Academy of Music, R&D Dept. Academy of Art & Design, IxDM School of Engineering, 4D Institute Multitouch Multitouch Soundmanipulation Virtual & Augmented Soundspaces Ludosonica Sonic Classroom Das klingende Klassenzimmer Sonically Augmented Playgrounds Computational Music Computational Music Thinking (SI EduNAT) Sonification of Mathematical Phenomena (EMP-Math) Sonification of Crawl Swim Movement Physical Sound Computing Sonically Augmented Playgrounds 30.04.2015 6 3
FOA 1 // Multitouch Sound Manipulation Abundance of music apps the common basics, elements of music education (e.g. harmony, composition, performance) is covered (in principle) Major paradigms: mimetic approaches, skeuomorphism embodiment of musical theory, harmony etc., often combined with prefabricated elements or automatisms Garageband: Sequencer & mimetic (smart) instruments, skeuomorphism Rockmate: Mimetic, keyboard paradigm Thumbjam: More abstract, embodiment of musical systematics, sensors add «expressivity» 30.04.2015 7 FOA 1 // Multitouch: Old Wine in New Wineskins? Pitch Painter: Yet another piano roll Bubl Draw: A fancy loop sequencer no real connection between shapes and music (but very nicely crafted!) 30.04.2015 8 4
Multitouch Pen-On-Paper Mockup Experiment TAIK Group 4 (SDS) TAIK Group 3 (SDS) PH-FHNW, Group 5 (MES) EAS, Group 3 (ME) 30.04.2015 11 FOA 2 // Soundspaces & FOA 4 // Physical Sound Computing: Sonic Playground Layla Gaye et al., Sonic City Examples of Sonic Playground Prototypes from Workshop at Interaction Design, ZHdK (Hug & Kemper) 30.04.2015 16 5
FOA 2 // Soundspaces: Sonic Classroom Ludosonica InHands System, based on PESI (http://sopi.media.taik.fi) 30.04.2015 17 FOA 3 // Computational Music Thinking, EMP Math & Music PingPong Simulator, Martin Guggisberg, 2015 AgentCubes (A. Repenning, PH FHNW Prof. Informatische Bildung) 30.04.2015 18 6
Auditory Display & Sonification Precursors: Earcons & Auditory Icons Listening to the Mind Listening: Sonification and Music Example by John Dribus Seibert: Movement Sonification 30.04.2015 19 Literature Bauer (2014): Music Learning Today: Digital Pedagogy for Creating, Performing, and Responding to Music, Oxford University Press: New York Buchenau & Suri (2000): Experience Prototyping, Symposium on Designing Interactive Systems, p424-433 Brandes (2008): Design durch Gebrauch: Die alltägliche Metamorphose der Dinge. Birkhäuser Clarke (2005): Ways of Listening An Ecological Approach to the Perception of Music. Oxford University Press Cross (1982): Designerly Ways of Knowing, Design Studies, 3, 4 Haverkamp (2012): Synesthetic Design Handbook for a multisensory approach. Birkhäuser Hug, Daniel (2007): Game Sound Education at ZHdK - Between Research Laboratory and Experimental Education. In Proceedings of Audio Mostly 2007-2nd Conference on Interaction with Sound (pp. 168 173). Ilmenau: Fraunhofer IDMT. Hug, Daniel (2010a): Investigating narrative and performative sound design strategies for interactive commodities. In Sølvi Ystad, Mitsuko Aramaki, Richard Kronland-Martinet and Kristoffer Jensen (eds.), Auditory Display volume 5954 of Lecture Notes in Computer Science (pp.12 40). Heidelberg: Springer. Hug & Kemper (2014): From Foley to Function: A Pedagogical Approach to Sound Design for Novel Interactions. Journal of Sonic Studies, volume 6, nr. 1 (January 2014) Kaptelinin & Nardi (2006): Acting with Technology. Activity Theory and Interaction Design Krebs (2014): Musikinstrumente im Taschenformat, in: nmz 02/2014 Mareis, Held, Joost (2013); Wer gestaltet die Gestaltung? Praxis, Theorie und Geschichte des partizipatorischen Designs. transcript. Miranda & Wanderley (2006): New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. A-R Editions Norman (1988): The Design of Everyday Things. Basic Books Riley (2013): Teaching, Learning, and Living with ipads, in: Music Educators Journal vol. 100 Smalley (1997): Spectromorphology: Explaining sound-shapes, Organised Sound: Vol. 2, no. 2. Cambridge: Cambridge University Press: 107-126. 30.04.2015 20 7
Thank You! Interested in exchange or collaboration? -> mail me: daniel.hug@fhnw.ch Daniel Hug School of Education, University of Applied Sciences & Arts of Northwestern Switzerland April 24, 2015 8