Development of extemporaneous performance by synthetic actors in the rehearsal process Tony Meyer and Chris Messom IIMS, Massey University, Auckland, New Zealand T.A.Meyer@massey.ac.nz Abstract. Autonomous synthetic actors must invent variations of known material in order to perform given only a limited script, and to assist the director with development of the performance. In addition, the synthetic actors need to learn through the rehearsal process, as their human counterparts do, via feedback from the director. Through the production of two performances, involving both human and synthetic actors, a variety of methods of creating extemporaneous performance and utilising feedback to select the most effective performance variations will be examined. One method of varying the performance is the manner in which lines of dialogue are delivered. The paper outlines use of a statistical technique to create three variations of a performance; each variation was then ranked, and these rankings used to weight the variances in individual lines to create a superior variation. This allowed quick evaluation of many lines, without having to score each individual line. Synthetic Actors Synthetic actors and computer theatre Computer theatre is about providing means to enhance the artistic possibilities and experiences of professional and amateur actors [1]; we are concerned with live, real-time, theatrical performances that use computers in the artistic process. Historically, these characters are typically (especially the most well-known examples) only electronic puppets, whose behaviour is directly simulated by a puppeteer, although recently extra characters in films have been autonomous synthetic actors (for example, in Lord of the Rings: Return of the King, controlled by the MASSIVE software [2]). It is the category of autonomous synthetic actors that is the authors focus. To interact with the human actors, the synthetic actors must possess a variety of actuators and sensors, such as audio and visual recognition and generation [3]. The project currently includes speech synthesis and recognition, a graphics generator, and a vision system. The speech systems use the Microsoft Speech SDK; the graphics generator uses the 3D engine from Unreal Tournament 2003, via the Gamebots [4] module (see
2 Tony Meyer and Chris Messom Figure 1). The design of the project allows other sensors and actuators to be added as necessary; for example, for a brief scene requiring physical interaction between human and synthetic actors in The Porcelain Salamander (described below) a very simple robotic actuator will be utilised. Fig. 1. Synthetic actors as generated images Aims Improvisation The aim of an improvised theatrical performance (whether part of a rehearsal, or otherwise) is to appear to play extemporaneously by inventing variations with whatever materials are at hand [5], rather than to perform with little or no preparation [5]. Anderson and Evens define improvisation as creating minor to extensive variations on a routine in a satisficing manner in real time [6]. While autonomously creating a believable and dramatically valid theatrical performance with no preparation may be beyond synthetic actors, the equally valid task of creating interesting variations of known behaviour is certainly not. Improvisation plays an important role in theatrical performances, even those comprised of scripted dialogue. Improvisation fills the gaps left by the script (most movement and how dialogue should be expressed); essentially, the playwright provides a task, but it is up to the actor to decide how to execute that task to maximise the enjoyment of the performance by the audience. In addition, although the playwright and director both begin the rehearsal process with a particular expression of the performance in mind, this is refined during the rehearsal process, typically through minor alterations of dialogue or of the actor s perception of their character. The way in which the actor decides to improvise their performance contributes to this process, by expanding the range of possibilities visible to the director.
Development of extemporaneous performance by synthetic actors in the rehearsal process 3 Learning The other key role of the rehearsal process is practice: sections of the play are performed, feedback is gained from the director (and possibly other actors), and, using that feedback to alter the performance, the section is performed again, until an optimum performance is achieved. Typically each section will be rehearsed several times per rehearsal, then a break is taken (during which the actor is expected to consider methods of further improving their performance), and then the section is again rehearsed at the next rehearsal. Current synthetic actors tend to learn outside of this process a director will review the performance by the synthetic actor, usually after the human actors performances are finalised, and alter it until it fits with the performance. This typically involves manually altering parameters of the performance or introducing random elements into the performance (as the human component of the performance is unable to be changed). A more balanced development, and more truly autonomous synthetic actors, would result from human and synthetic actors together modifying their performances to suit each other, as the rehearsal process progresses. A major difficulty that arises is that the feedback given to each actor during the rehearsal process is very limited in particular, only a few attempts at each scene can be made; actors are expected to extrapolate an understanding of what is required from this and use that to evaluate possible performance improvisations (outside the rehearsal arena). Methodology To test various methods of generating improvisational performances and learning through the rehearsal process, two live stage performances featuring synthetic actors will be produced. Each production will include a small number of human actors acting alongside a small number of synthetic actors, who will play roles that a human actor would have difficult portraying (typically as a result of physical constraints). Throughout the production, data will be collected and analysed, through quantitative methods and phenomenological analysis of qualitative data. The first of these productions, scheduled to be performed in late June 2004, will be an adaptation of the short story The Porcelain Salamander (by Orson Scott Card), featuring four human actors and one leading and three minor synthetic parts. The story is dramatic (essentially a fable), rather than comedic; the fantasy setting of the story is well suited to experimental theatre, and will assist the traditional theatrical suspension of disbelief. The magical and slightly unreal nature of the character and story also provides some leeway with the level to which the speech and graphics must appear realistic. The salamander character is well suited to a synthetic actor, as it is not alive, but has been given the gift of movement and speech [7], much like synthetic characters themselves. The character is relatively simple, but is a lead, so still requires a level of
4 Tony Meyer and Chris Messom performance above that of simple scripting. The character behaves differently depending on which other (human) characters are currently on stage, and moves through various emotional states (particularly happiness, fear and sadness). For a successful performance, the audience must understand how one of the other (human) characters could come to love the salamander character, and the salamander s final sacrifice must draw an emotional response from the audience. Preliminary Results Dynamic selection of affect for generated speech The synthetic actors have the same script available as the human actors, which includes all the words to be spoken, but almost no information about how the lines should be delivered. A human actor uses a variety of techniques to decide on the appropriate delivery of each line [8], including analysis of the context of the line, the words in the line itself, and their perception of the character they are playing and the state of the other characters. The synthetic actor begins without a model of the character (but may have generic knowledge from a prior performance), so must initially rely solely on the lines of dialogue. Classification of dialogue by affect 500 Lines of dialogue 400 300 200 100 0 1st 2nd 3rd Unsure Incorrect Fig. 2. Cross-validation results from classifying lines of dialogue into affect categories. Seven basic methods of dialogue delivery were implemented, mimicking the basic emotions of happiness, sadness, anger, disgust, and fear [9], along with surprised and neutral unaltered speech. For each line of dialogue a score for each emotion was determined using a statistical technique [10]; the strongest scores were combined to determine the desired delivery. To vary the performance, the synthetic actor was able to alter the combination of scores (e.g. use just the strongest score, or combine the two strongest scores). To learn from the performance, the synthetic actor trains on each line of dialogue, indicating that it matches one or more of the delivery methods. Even without any training on the current performance, results are satisfactory, with
Development of extemporaneous performance by synthetic actors in the rehearsal process 5 around 45% of lines correctly delivered and only around 16% of lines unable to be delivered correctly given three attempts (see Figure 2). Application of director feedback In one rehearsal, four attempts at each scene involving the salamander character, using a different delivery method for each line (generated as above), were carried out. With each of the first three attempts, each of the five scenes (each comprised of between five and eighteen lines by the synthetic actor) was rated by the director with a simple score, with ten corresponding to a performance where all lines were well delivered, and zero corresponding to a performance where every line should have been delivered differently. These three attempts were independent (i.e. no training was done between these attempts), with the first attempt ranking highest and the second attempt lowest. The fourth attempt used these scores to combine the affect tags for each of the three performances to deliver a final performance superior to all three prior performances (see Figure 3). Perform ance Rating Total Rating 50 40 30 20 10 0 Attempt 1 Attempt 2 Attempt 3 Attempt 4 (Combined) With 5 explicit corrections Scene 5 Scene 4 Scene 3 Scene 2 Scene 1 Fig. 3. Performance ratings (by the director) of three generated attempts and two attempts utilising feedback. To combine the affect tags, the system weighted the score of the tag for each attempt with the rank given to that scene and selected the tag with the highest weighted result. This meant that where the system was much more certain that it was correct, the choice of tag would not necessarily be simply the one from the attempt with the highest rank. The fourth (combined) attempt therefore combined the strengths of each of the three previous attempts line-by-line, rather than simply sceneby-scene, as it was ranked. This did not allow for the fourth attempt to use any tags not used in any of the three previous attempts, however, and so around 10% of lines (scattered throughout the scenes) still failed to be delivered as effectively as possible. This technique allows quick evaluation of many lines, without the director having to score each individual line; when combined with the ability to explicitly instruct the
6 Tony Meyer and Chris Messom actor that a particular line (delivered poorly compared to the others in the scene, or delivered poorly in all previous attempts) should be delivered in a particular emotion, successive performances rapidly improved (see Figure 3). In the future, more sophisticated methods of combining this data to select more effective improvisation choices will be evaluated. Conclusion Dramatically effective extemporaneous performance within a scripted performance is an achievable goal for autonomous synthetic actors. Although the rehearsal process only offers a limited amount of feedback, the synthetic actors are, like their human counterparts, able to extract sufficient training data from this feedback to learn which forms of improvisation are particularly effective for each component of the performance. Through the production of two live stage performances with a cast including both human and synthetic actors, various techniques of extracting this data, and applying it to the development of a character model, will be fully evaluated. References 1 C. Pinhanez, "Computer Theatre," M.I.T. Media Lab, Perceptual Computing Section Technical Report 378, 1996. 2 "MASSIVE", [online] 2004, http://www.massivesoftware.com/index.html (Accessed: 11 March 2004) 3 T. A. Meyer and C. H. Messom, "Development of Computer-Actors Using the Interval Script Paradigm," presented at 9th Annual New Zealand Engineering & Technology Post Graduate Conference, Auckland, New Zealand, 2002. 4 R. Adobbati, A. N. Marshall, A. Scholer, S. Tejada, G. A. Kaminka, S. Schaffer, and C. Sollitto, "Gamebots: A 3D Virtual World Test-Bed for Multi-Agent Research," presented at Second International Workshop on Infrastructure for Agents, MAS, and Scalable MAS, Montreal, Canada, 2001. 5 Houghton Mifflin Company, The American Heritage Dictionary of the English Language, 4th Ed. ed: Houghton Mifflin Company, 2000. 6 J. Anderson and M. Evens, "Constraint-Directed Improvisation," presented at Eleventh Biennial Canadian Society for Computational Studies of Intelligence Conference, Canada, 1996. 7 O. S. Card, "The Porcelain Salamander," in Unaccompanied Sonata and Other Stories: Dial Press, 1981. 8 G. Colson, Voice Production and Speech. London: Museum Press Limited, 1963. 9 A. Ortony and T. J. Turner, "What's Basic About Basic Emotions?," Psychological Review, vol. 97, pp. 315-331, 1990. 10 T. A. Meyer, "Dynamically Determining Affect during Scripted Dialogue", [online] 2004, http://www.massey.ac.nz/~tameyer/research/computertheatre/docs/tameyerpricai.pdf (Accessed: March 8 2004)