INTERACTION. Basics of Real-Time Interactive Audio Processing. Real-time Interaction with Recorded Sounds

Transcription

1 Basics of Real-Time Interactive Audio Processing Real-time Interaction with Recorded Sounds DAAD Edgar Varèse Guest-Professorship 2007 TU Berlin Audio Kommunikation INTERACTION Norbert Schnell Berlin, May 24 & 25, 2007

2 chapter 0 interaction concepts

3 PCI performer-computer interaction composer, artist performer, user instrument, interface? machine?????

4 multi-modal interaction capture input devices (mouse, keyboard, faders,...) sensors video audio + movement and gesture analysis/recognition + computer vision + audio and music analysis/recognition feedback audio (auditive) visual haptic

5 discourses & levels elements of interaction interaction metaphors interaction paradigms models & representations techniques applications

6 chapter 1 elements of interaction

7 elements of interaction object instrument sculpture box thing machine

8 elements of interaction object - space - room stage wall yard town

9 elements of interaction object - space - image object material space mirror recall impression illusion map shadow appearance imagination

10 material? (surface?, texture?) active surfaces and textures wearables interactive fabrics etc...

11 chapter 2 metaphors and paradigms

12 metaphor vs. paradigm metaphor dict.: a figure of speech in which a word or phrase is applied to an object or action to which it is not literally applicable analogy ideal paradigm dict.: a typical example or pattern of something; a model operational or even established way of doing standard

13 music related metaphors playing an instrument playing with accompaniment conducting an orchestra one-man-band ensemble playing (duo, trio, many) mixing and DJing listening to music

14 extra-musical metaphors communication conducting navigation games and puzzles sports and competition combat writing and desktop work life (biological processes, social behaviour)...

15 paradigms & stereotypes performance and expression the music player automatic accompaniment the continuator simulation and control active (enhanced) listening real-time interactive content-based audio proc. visualisation & sonification

16 chapter 3 models and representations

17 model vs. representation (don t mind :-)... vs. description

18 IwCS* models + devices sound model + sound rendering system gesture/movement model + interface, capture device mapping between sound and gesture model (correspondence, coincidence, adequacy,???) + overall interaction design * Interaction with Captured Sounds

19 sound model* model of a temporal object already there, ready made, completely defined, deferred modelling = conditioning frozen, solid liquid, fluid, supple, elastic, articulated representation encoding description (characteristics, morphology, semantics, etc.) decomposition interaction * recorded sound (vs. captured sound)

20 gesture/movement model model of a process action, stream, real-time modelling = analysis & recognition shadow, projection of an action intention, interaction representation measurement (interface parameters) projection reconstitution (of original action and intention) decomposition semantics, symbolics, intention sound and music

21 sound representations waveform spectrogram & co (linear/logarithmic, continuous/quantised) sonogram, cepstrum, MFCC, X-Micks, chromagram,... harmonics and partials audio features (instantaneous and dynamic) modulation spectrum segmentation similarity matrix score piano roll...

22 gesture interfaces (= representations) keyboard and mouse (with or without GUI) (graphic) tablet faders and knobs MIDI keyboard physical sensors accelerometer, giroscope, flex, pressure, distance, etc. computer vision (EyesWeb & Co) object and motion descriptors 3D motion capture audio analysis...see NIME & Co

23 chapter 4 examples

24 performance interfaces NIME IwCS performers Michel Waisvisz The Hands Atau Tanaka Biomuse Serge Delaubier MetaInstrument erikm 3-k.pad.system Tomomi Adachi - Voice and Infrared Sensor Shirt Sven König scrambled?hackz! (see below: corpus based) more interfaces ReacTable Sergi Jorda et al.... other NIME 06 performances/interfaces

25 more artists live FX Joel Ryan (esp. with Evan Parker) Olvier Sens (Usine) composition and IwCS Philippe Manoury En Echo, Partita 1 Florence Baschet Bogenlied Thierry DeMey Light Music Pierre Jodlowski Le Grand Dehors

26 installations (examples) Listen Gerhard Eckel et al. 16:9 Daniel Teige & Martin Rumori Audio Shaker Mark Hauenstein (... more???!!!)

27 gesture analysis/recognition software EyesWeb A.Camuri et al. (Univ. Genova GIST) cv.jit Max/MSP/Jitter computer vision tools IAMAS MnM Max/MSP/FTM toolbox IRCAM IMTR gesture follower F. Bevilacqua et al. (IRCAM IMTR) dance projects one video example: Olivia Granville comment taire many many projects!!!???

28 applets ENIGMES: Feuillage Eric Broitmann et al. IXI Software

29 generative systems The Continuator - F. Pachet OMax G. Assayag, G. Bloch et al. Khorwa M. Malt (... too many missing)

30 MIR / corpus based concatenative/corpus based synthesis systems catart Diemo Schwarz (IRCAM IMTR) SoundSpotter Michael Casey bbcut Nick Collins song segmentation/résumé sonore G.Peeters et al. (IRCAM) performances and installations scrambled?hackz! Sven König messa di voce Golan Levin and Zach Lieberman

31 Varèse SS 2007 Hausaufgabe zum 25.Mai 1-5 Klänge auswählen (mitbringen) Beschreibung ihrer Morphologie Vorstellung einer möglichen Interaktion Finden/Auswählen bis zum 14.Juni IwCS Paradigma Schnittstelle Klangverarbeitungsumgebung Wilms Studio Di 5. & Fr 8.Juni ab Uhr Arbeitsgruppe