Real-Time Interaction Module

Transcription

1 Real-Time Interaction Module Interdisciplinary Master in Cognitive Systems and Interactive Media Session 4: On Mapping Prof. Sergi Jordà

2 Index Part I Introduction Mapping definitions Convergent, divergent & complex mappings A case study on mapping Flow in HCI Part II Complex mappings Mathematical formulations Part III RTI Conclusions & Future

3 Part I: the Human Side

4 Mapping / What? Physical and logical separation of the input device from the sound production, brings the necessity to process and map the information coming from the input device in a variety of ways Mapping studies the logical connections between gestural parameters (or any other kind of input data), and output parameters/results Data Mapping (Wikipedia def. not related to interaction)

5 Data mapping (from Wikipedia) Data mapping is the process of creating data element mappings between two distinct data models. Data mapping is used as a first step for a wide variety of data integration tasks including: Data transformation or data mediation between a data source and a destination

6 Mapping / When? When not? (1/2) When the coupling of the input and output parameters is direct (e.g. a touch screen, a glove for handling virtual objects in VR, a HMD in VR ) mapping is invisible (1:1) In indirect pointing devices mapping is often only a matter of scaling e.g. Mouse: x it x ot, y it y ot x o = a x i x o t = a x i t + b * ( x i t x i t-k ) Idem (scaling) for indirect 1D devices, or indirect devices connected to the real world e.g. turning in a steering wheel Teleoperation/microsurgery

7 Mapping / When? When not? (2/2) When the interaction metaphor is very clear, mapping can also be quite straightforward e.g. Michael Lyon s Mouthesizer the same applies to digital interfaces that mimic existing devices mapping becomes a matter of fine tuning In more abstract interfaces, mapping is not so straightforward

8 The Yamaha WX5 wind controller: a Case Study The Yamaha WX5 wind controller detects wind pressure and lip pressure (+ discrete fingering) usually wind pressure is interpreted as loudness and lip pressure is interpreted as pitch bend, yet. things may not be so simple (Rovan 97) x oi = f(x i1 x in )

9 Traditional Instruments Behavior While an engineer looks after efficiency, non-ambiguity thus avoiding non-linear, correlated controls many musical instruments do have a very bad design from this point of view: In many instruments (e.g. trumpet & other brass), loudness and pitch are slightly correlated (i.e. they are not completely independent) In the voice, this correlation is much more important It even seems that it is within these dependencies that lies part of the potential expressiveness of these instruments

10 One-to-one, one-to-many, many-to-one x i y i one-to-one what does the bow of the violin control? one-to-many where is the violin volume control? many-to-one

11 One-to-one, one-to-many, many-to-one x i y i one-to-one what does the bow of the violin control? one-to-many where is the violin volume control? many-to-one many-to-many

12 Mapping Studies How to define a good mapping? Difficult to abstract from the choice of input devices Only very simple cases, tend to be studied systematically, with few input & output parameters Yet, it seems that less obvious mappings often feel and work better Effort and difficulty is not always bad, when looking for expressive control/interaction

13 Hunt et al A series of tests was carried out at the University of York UK in order to study the effectiveness of different interfaces when used for a real-time musical control task. The data that was gathered was used to compare how a group of human test subjects performed on different interfaces over a period of time. The following three interfaces were chosen for the study: 1. A set of on-screen sliders controlled by a mouse 2. A set of physical sliders moved by the user's fingers 3. A multiparametric interface (mouse and one hand + two sliders in the other) The goal was controlling, volume, pitch, timbre and panning on a synthetic sound Different mappings were applied The 3rd interface was preferred, although preference depended very much on the mapping (more than on the control interface)

14 Hunt et al (some conclusions) Volume needs energy (speed) + button pressed allows quick changes and attacks average position introduces some variation/instabilities Volume dx/dt (e.g. speed of a slider) (wind sensor, pressure sensor ) Also users learned further with more complex mappings! Volume speed of mouse + mouse but. pressed + average position of 2 sliders Pitch vertical position of the mouse + speed of movement of slider no. 2 Timbre Horizontal position of the mouse + difference in the two slider posit.

15 Hunt et al (some conclusions) Volume needs energy (speed) + button pressed allows quick changes and attacks average position introduces some variation/instabilities Volume dx/dt (e.g. speed of a slider) (wind sensor, pressure sensor ) Also users learned further with more complex mappings! Volume speed of mouse + mouse but. pressed + average position of 2 sliders Pitch vertical position of the mouse + speed of movement of slider no. 2 Timbre Horizontal position of the mouse + difference in the two slider posit.

16 Hunt et al (some conclusions) Vertical position: higher/lower obvious Speed of movement raises pitch when more energy Volume dx/dt (e.g. speed of a slider) (wind sensor, pressure sensor ) Also users learned further with more complex mappings! Volume speed of mouse + mouse but. pressed + average position of 2 sliders Pitch vertical position of the mouse + speed of movement of slider no. 2 Timbre Horizontal position of the mouse + difference in the two slider posit.

17 Hunt et al (some conclusions) The results may seem/be a bit arbitrary Other mappings could have been tested But the preferred one still depict some [at least domain specific/ musical control] logic Could we also extract some general/non-domain-specific conclusions or guidelines? (Hunt et al. 2000) Mappings which are not one-to-one are more engaging for users. One-to-one mappings exhibit a toy-like characteristic. Complex tasks may need complex interfaces. Simplicity/easiness may not always be the best choice?

18 Flow (Csikszentmihalyi 75,90) Flow is the mental state of operation in which the person is fully immersed in what he or she is doing by a feeling of energized focus, full involvement, and success in the process of the activity. Proposed by positive psychologist Mihály Csíkszentmihályi, the concept has been widely referenced across a variety of fields. Colloquial terms for this or similar mental states include: to be on the ball, in the zone, or in the groove. Cs%C3%ADkszentmih%C3%A1lyi

19 Flow (Csikszentmihalyi 75,90) Csíkszentmihályi identifies the following 9 factors accompanying an experience of flow: 1. Clear goals (expectations and rules are discernible and goals are attainable and align appropriately with one's skill set and abilities). 2. Concentrating and focusing, a high degree of concentration on a limited field of attention (a person engaged in the activity will have the opportunity to focus and to delve deeply into it). 3. A loss of the feeling of self-consciousness, the merging of action and awareness. 4. Distorted sense of time, one's subjective experience of time is altered. 5. Direct and immediate feedback (successes and failures in the course of the activity are apparent, so that behavior can be adjusted as needed). 6. Balance between ability level and challenge (the activity is neither too easy nor too difficult). 7. A sense of personal control over the situation or activity. 8. The activity is intrinsically rewarding, so there is an effortlessness of action. 9. People become absorbed in their activity, and focus of awareness is narrowed down to the activity itself, action awareness merging.

20 Csikszentmihalyi 90 Csikszentmihalyi covers flow in contexts such as: Art & creation Music performance Sports practice & martial arts Scientific research Sensing (listening to music, eating.) Human relations Work (autotelic work).. Although he does not cover HCI, his work has been quite influential in the field

21 Flow in HCI Creativity, playfulness, engagement The experience of flow in computer-mediated and in face-toface groups (Ghani et al. 91) FlowTheory.com An MSc dissertation exploring the importance of website usability from a business perspective; contains a section on flow. Flow in videogames (Chen 06)

22 Part II: Complex mappings

23 Mathematical Formulations In Maths, mapping is often a synonym for function thus the simpler mapping: x I, y / y = f ( x) Problems/limitations Does not guaranty that all elements of O are the image of some function f that could take one or several (many-to-one) elements from I some elements from the generator could not be attainable by the controller Does not contemplate one to many (the relation would not be a function anymore) Goudeseune 2002, Van Nort et al have tried to integrate more complex mappings into a complete mathematical formulation success unclear for several reasons

24 Mapping and memory Mapping functions are in most cases implicitly assumed to be instantaneous, yet, keeping track of the history would permit more musical behaviors: Smoothing responses (low-pass filter), amplifying changes (highpass filter), modifying the instrument responsiveness in different manners (using output and/or input history) Measuring input speed (a differentiator using input history) Measuring average activity Detecting gestures (using input history) Memory y it = f(x it.x it-k ) Feedback y it = f(x it.x it-k,y it-1.y it-k ) NM y it = f(x 1t.x Nt ) and any combination of them (equation of a recursive IIR filter)

25 Additional (and realistic) mapping possibilities (Ryan 91) Shaping Response Controller data can be shifted or inverted [addition], compressed and expanded [multiplication], limited, segmented or quantized [thresholding]. Methods which keep track of the history of a signal allow measurement of rates of change, smoothing and other types of filtering to amplify specific features in the signal or to add delays and hysteresis in the response [differencing, integration, convolution]. The rates of data transmitted can be reduced and expanded [decimation and interpolation]. Linear and nonlinear transforms allow the shaping or distortion of the signals to any desired response [functional and arbitrary mappings] (Ryan 1991). Feedback also allows for the appearance of chaotic and selforganizing behaviors

26 [insert] Non-linearity, feedback & expressivity (Jordà 05) Many traditional instruments can be driven to a state of chaotic behavior characterized by noisy, rapidly fluctuating tones. Examples could be found in the vocalized saxophone style in which vocal sounds interact directly with vibrations in the instrument (Menzies 2002). The use of feedback in the electric guitar converts the guitar into an element of a complex driven system, where the timbral quality and behavior of the instrument depends on a variety of external factors, including the distance to the loudspeakers, room acoustics, body position, etc. Musicians explore and learn to control these additional degrees of freedom, producing the very intense, kinetic performance styles upon which much of free jazz and rock music is based. If non-linearity is at first intuitively seen as a source of potential uncontrol, it can therefore also mean higher-order and more powerful control. Distinct virtuosity paradigms definitely coexist: whereas the classical virtuoso, with his infinite precision and love for details may appear closer to the goldsmith, the new digital instruments virtuoso, not unlike the jazz one, could be compared to the torero for his abilities to deal with the unexpected.

27 Few to Many An added problem is that often, the output dimension is far superior than the controller dimension few-tomany mapping (Lee & Wessel 92) In the music (NIME) field, many authors have proposed to decompose the mapping chain into various levels, from gestural to perceptual and from perceptual to synthesis parameters

28 Lowering perceptual dimensionality (in gesturalperceptualsynthesis) Reducing how many dimensions of control an instrument has, makes it less frightening to its performer. More formally, such a reduction concentrates the set of all possible inputs into a more interesting set by avoiding the redundancy inherent in the exponential growth of increasing dimensionality. Even more formally, it reduces the dimensionality of the set of synthesis parameters to the dimensionality of the set of perceptual parameters: it rejects all that the performer cannot actually understand and hear, while performing Controlled loss of information is about discovering what the performer can and cannot do, about matching that dividing line with the one between expressive and inexpressive (Goudeseune 2002).

29 Multilayered Perceptive Mapping (Arfib) for better mapping intention to sound expression

30 Automating mapping? Made-to-measure solutions may work for each individual case, but couldn t we propose a more generic approach to this problem? MN data mappings and dimension reduction, are very common problem nowadays. E.g: Plotting data into 2 dimensions from a high dimensional original space is a hot topic on artificial intelligence, data visualization, data exploration and document retrieval. In Music Information Retrieval (and other IR areas), plotting a set of M computed tags (energy, centroid ) to a set of N perceptive tags (danceability, happiness ) Several techniques are currently used Linear projections (plus Principal Component Analysis techniques) Self-Organizing Maps (neural networks for unsupervised learning) Dynamic Bayesian Networks Autoencoders (neural networks)

31 Linear projections allow us to represent elements from a high-dimensional space projecting them perpendicularly in a subspace with less dimensions. Finding this subspace to loose the minimum information is the heart of the matter. To accomplish it, we can apply techniques such as Principal Component Analysis (PCA), which finds the nths first directions that have the maximum variance. Self-Organizing Maps are neural networks used for unsupervised learning which try to preserve the original topology of the former space in a projected lowdimensional space. The points that close together on the final space also have to be close on the original one. The resulting transformation is not linear, so it cannot be matched to a typical subspace. It is also considered a clustering method. SOM is an iterative process. Dynamic Bayesian Networks, use a probabilistic model based on an uncyclical graph whose nodes have random variables within, and it obeys the Bayes Theorem. It is very useful to discover the hidden causes of the data. Autoencoders are a special type of neural networks. These have an automatic system for the maximization of the information in the resulting dimensions. This is done by building a decoder which tries to reconstruct the original data space from the resulting low-dimensional data. The decoder and the encoder (the part of dimensional reduction) are adjusted and, if we get results similar enough to the former data, we are guaranteeing the high relevance of the low-dimensional data.

32 Unsupervised mapping In Music Information Retrieval (and other IR areas), plotting a set of M computed tags (energy, centroid ) to a set of N perceptive tags (danceability, happiness ) Autonomous robots, plotting a set of M sensor inputs to a set of N actuators (e.g. An artificial moth: Chemical source localization using a robot based neuronal model of moth optomotor anemotactic search, Pyk et al. 06, SPECS) These problems seem quite similar to the previous figure Yet, systematic research in that direction does not seems to have been taken yet

33 References Arfib, D., Couturier, J.M., Kessous, L., and Verfaille, V. (2002). Strategies of mapping between gesture data and synthesis model parameters using perceptual spaces. Organised Sound 7, Csikszentmihalyi M. (1990), Flow: the Psychology of Optimal Experience. Harper Perennial Goudeseune, C (2002). Interpolated mappings for musical instruments. Organised Sound, 7(2), Hunt, A., & Kirk, R. (2000). Mapping Strategies for Musical Performance. In M. Wanderley and M. Battier (eds.), Trends in Gestural Control of Music. Paris: IRCAM - Centre Pompidou. Hunt, A., Wanderley, M. M. & Kirk, R. (2000). Towards a model for instrumental mapping in expert musical interaction. In Proceedings of the 2000 International Computer Music Conference. San Francisco, CA: International Computer Music Association, Rovan, J., Wanderley, M. M., Dubnov, S. & Depalle, P. (1997). Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance. In Proceedings of the Kansei - The Technology of Emotion Workshop, Genova - Italy, Oct Hunt, A. (1999). Radical User Interfaces for Real-time Musical Control. Ph.D. thesis, University of York UK. Ryan, J. (1991). Some Remarks on Musical Instrument Design at STEIM. Contemporary Music Review 6(1), Ryan, J. (1992). Effort and expression. In Proceedings of the 1992 International Computer Music Conference. San Francisco, CA: International Computer Music Association,

34 Part III : RTI Overview and Conclusions

35 RTI Overview We have first tried to define concepts such as Interaction and Real-time, in order to deduce what RTI is/could be We have then enumerated some of the areas in which RTI is relevant (videogames, VR, AR, interactive installation, music performance) We have studied a quick historical and conceptual overview of Interactive Music Systems, in order to grasp its endless possibilities, and as an example of a potentially very complex interaction domain We have studied input controller devices We have studied mapping, for understanding how to connect input devices with interactive digital systems

36 RTI Discussion Although RTI concepts have not been yet very relevant in productivity or offimatic related areas, this trend may be changing Several reasons? Cooperative work (CSCW) and Shareable interfaces (collocated collaborative interfaces; e.g. tabletop interfaces) simultaneous and multidimensional input Data explosion data mining & data visualization interactive data exploration / exploratory search maximizing interaction bandwidth (new interfaces can reduce the cognitive load or the perceptual distance between the researcher and the data).

37 HCI has mostly focused on making things more efficient, not discovering until apparently quite recently that not only efficiency is relevant, but perhaps also beauty and fun (e.g. Norman, 2004; McCarthy and Wright, 2004). HCI also devotes much effort in making task-solving simpler, often forgetting Albert Einstein s famous apocryphal quote everything should be made as simple as possible, but no simpler. Computers were not only made for making simple things, simpler, but also for making impossible things, possible (e.g. Engelbart, 1962). When complex and rich interactive processes are oversimplified, by definition, control over them can only be degraded or lost. As Buxton (1997) points out, it seems that computer devices and tools have almost never been conceived for the skilled user in mind. The study of music performance not only opens our eyes (and our ears!) to the marvels of skilled performance; it reminds us that complex interaction can also be enormously fun and rewarding for everyone, even novices,... and that we should not be scared of complexity, when such complexity is needed. (Jordà 08)

38 Music performance and control can constitute an ideal source of inspiration and test bed for exploring novel ways of interaction, especially in highly complex, multidimensional and continuous interaction spaces such as the ones present when browsing huge multimedia databases. This type of interaction involves searching in complex, hyperpopulated and multidimensional spaces, often looking for unknown and probably not single targets; a process that could be better compared with playing a violin that being reduced to the six generic virtual input devices that constitute the GKS standard. In these types of fuzzy interaction environments, exploration can follow infinite paths, results can hardly be totally right or wrong, and the evaluation metrics can become so unclear, that joyful and creative use may become one of the essential assets (Jordà et al. 07). Creativity is present not only in the production of nice pictures or music, but specially in the process of creation or construction of anything. Creativity, related with expressiveness and with freedom, can thus become important in any interaction process enough complex or enough free, such as the ones in which the paths to a goal are open or when the goal itself is open (Jordà et al. 07).

39 Song Explorer: A case study (Song Explorer Video, Carles F. Julià 2008) Beats Per Minute (BPM) Happy probability Sad probability Party probability Acoustic probability Aggressive probability Relaxed probability

40 Other Topics that we did not cover Towards expert interaction: expressivity, non-linearity, control and virtuosity Interaction efficiency The frustration-boredom equilibrium Playability, explorability, progression and learnability Expressive and virtuosic interaction Complex non musical interaction Multipoint and multi-user interaction Multidimensional and continuous Interfaces for complex interaction Exploratory search: a case study Tabletop interfaces and the reactable: a case study Tabletop interfaces: multiuser, multipoint and tangible interaction Tabletop interfaces, multimodalism and bandwidth maximization The reactable as an interface for shared multithreaded control The importance of feedback

41 Other Topics that we did not cover Towards expert interaction: expressivity, non-linearity, control and virtuosity Jordà, S. (2005) Digital Lutherie: Crafting musical computers for new musics performance and improvisation, PhD. dissertation, Universitat Pompeu Fabra, Barcelona (chap. 7, pp ) Complex non musical interaction Tabletop interfaces and the reactable: a case study Jordà, S. (2008) On stage: the reactable and other musical tangibles go real, Int. J. Arts and Technology, Vol. 1, Nos. 3/4, pp Jordà, S., Geiger, G., Alonso, M. And Kaltenbrunner, M. (2007). The reactable: exploring the synergy between live music performance and tabletop tangible interfaces. Proceedings of the 1st international conference on Tangible and embedded interaction. Baton Rouge, Louisiana, Julià, C. F. (2008). Song Explorer: Exploring large musical databases using a tabletop interface. Master thesis, Universitat Pompeu Fabra. White, R.W., Kules, B., Drucker, S.M. and Schraefel, M.C. (eds.). Supporting Exploratory Search. Communications of the ACM, 49(4) (2006).