Generating Cinematic Camera Shots for Narratives Introduction John Mason CSC725: Intelligent Multimedia Systems Spring 2005 Arnav Jhala We have built a system that automatically generates camera actions for storytelling in virtual environments. Given a sequence of character actions, scene properties (like mood, tempo, etc, and a mapping from names to characters, objects, and locations in a graphical world, the goal of a camera planning system is to generate sequences of directives for the camera to satisfy derived from certain communicative goals that are input by the author. For this test case, we developed a specific implementation of the problem. For generating camera shot directives, we considered a library of character actions (specifically walk, say, and point/wave, attributes to annotate scenes, characters and stories, and communicative goals. For the demonstration we restricted ourselves to conversational settings within stories. Camera directives were generated by a planning system. The plan was then fed to a modified version of the Unreal Tournament 2003 game engine (hereafter as UT where it was executed. Our choice of conversational setting enables us to compare and contrast the effect of feeding different parameters to the planning system for the same basic scene. Approach Our approach involved the following steps: 1. Determining the parameters of the story, scene (situation, and characters that affect shot selection. 2. Analysis of various types of idioms used by cinematographers in films for conversational settings. 3. Selecting an appropriate situation from a story and annotating it with parameters for input to the camera planner. 4. Examining how our results differ, and hopefully improve upon, the existing situation. Determining parameters A narrative consists of the story and its discourse presented to an audience. The story can be represented as a series of actions and events occurring in the story world at different locations within that story world. Scenes are sequences of actions and events that are temporally and spatially adjacent in the story world. Characters and objects are also elements of a story. Characters and objects have various properties like mood, importance, attention, etc. Actions and events have types associated with them. The discourse of a story contains a telling of the story in various media. Properties of the story, like tempo (rate of temporal progression, affect how discourse is generated. Discourse of a narrative is also affected by the relationships between different story elements.
We developed several lists of parameter-value pairs. The earlier versions of the list included more, and more poorly defined, parameters. The lists were eventually distilled into a single useful form that was of manageable size and obeyed cinematic terminology and convention. Useful implies that the parameter-value pairs are in a form the planner can process and the output from the planner makes sense. The following table lists the parameters and their values for each cinematic element. Story Scene Action Character Camera Table 1: Parameters that affect shot selection Tempo Suspense Exceptions {?} Setting Intensity Tempo Ambience {formal, casual, tense, happy, sad} {vast, open, confined} Action_Type {say, think, argue, react, saywhilemoving, move, gesture, stand/sit, use_object, } Num_Participants {0, 1, 2, 3, N} Focus Secondary Mood Intensity C, set of characters, objects, actions or events C, set of characters, objects, actions or events {angry, sad, neutral, happy, nervous, afraid/surprised, romantic} Attention {focused, indifferent, other, distracted} Angle Distance FOV Type (for actors A, B, and X {hi, med, low} {extreme_cu, cu, mcu, ms, mls, ls, els} Automatically computed {OTS to actor X, Past A to B, Track, Zoom, 2-shot, 3-shot, N-shot, Establish, }
Analyzing films After some debate, we settled on exploring the conversational idiom because conversations are inherently affective exchanges, and we wanted to explore how a given scene might be altered if its emotive parameters were changed. An additional advantage to the conversational idiom is that actor movement is typically minimized, which is an advantage in our UT implementation because it drastically decreases the effort expended on making virtual actors perform physical motion. Once that decision had been made, we chose to gather our initial parameter-value data of conversational idioms by example. We analyzed movies with conversational sequences in different types of settings; movies that used setting to emphasize properties of the story, scene, or characters as mentioned in the previous section. For this purpose, we classified multi-person dialogue-intensive movies, or movies with such situations, into the following categories: - Action/Argument o A Few Good Men o Hulk - Party/Social o Rope o Life of David Gale o Midnight in the Garden of Good and Evil - Boardroom/Formal o 12 Angry Men - Group/Casual o Ocean s 11 o Ladykillers As it turns out, several of these movies have segments that fit into multiple categories, as was evidenced in our research. For instance, Rope has primarily a social setting, but includes some confrontational moments; A Few Good Men has a nice mix of courtroom action and formal meeting-room conversations; and Midnight in the Garden of Good and Evil has several courtroom and confrontational scenes. It s less important to pigeon-hole a film into a category, and more important to extract as much as possible from each scene. Some raw film data illustrating how we related the shot sequences from different settings in the movies we analyzed can be found in the attached excel files. We also extracted rules of thumb, tips, and idioms from popular film textbooks (Arijon, Mascelli. The idioms relevant to our implementation can also be found in the Appendix. Selecting situations After collecting the requisite information from the study of the aforementioned films, we selected two sequences from well-known theatrical plays. We decided to look at plays for several reasons. Theatrical sets are very precise, with set dressing placed for maximum affect; actors move only according to their blocking (a set of motion and emotive instructions given to them by the director, which is typically straightforward and therefore easy to replicate; plays are only ever viewed from the position of the audience (called the fourth wall in one long string of action, which through our system could be enhanced with different view angles and shot timing; and most importantly, plays are a prime example of the conversational idiom. We had intended to generate cinematic-style plans for these plays and see what they would look like if shot as a movie. On closer examination, Rope is an interesting combination of theater and film. While it is
literally a movie, it is shot in the style of a play utilizing a fourth wall and extended shot lengths with very few camera cuts. We thought it would be interesting and illustrative of the power of our system if we re-shot scenes from Rope as if it was a typical movie (various camera angles and positions, generally short shot lengths, etc. To that end we have included images of two kinds. The first are comparative shots; shots from the film and output from our system that replicates the same conditions. The second are shots unique to our system; shots, that given Rope s constraints, were impossible for Hitchcock. Implementation For the implementation we used an existing discourse planning algorithm developed by Jhala et al. [1] using the Mimesis Architecture described in detail in [2]. Due to the limitations of the algorithm we could implement a subset of the properties mentioned in Table 1. Figure 1 shows a snapshot of the input to the camera planner that is based on the play Dearly Departed. (character Royce (character Marguerite (object phone (location room_royce (location kitchen_marge (at Royce room_royce (at Marguerite kitchen_marge (mood Royce neutral (before s1 (mood Marguerite neutral (before s1 (conversation c1 (conv-type c1 phone (conv-start c1 s1 (convend c1 s2 (conv-steps c1 (s1 s2 s3 (step s1 (act-type s1 speak (agent s1 Royce (secondary s1 Marguerite (effect s1 (spoken Royce "What the... Hello" (mood Royce neutral (during s1 (step s2 (act-type s2 speak (agent s2 Marguerite (secondary s2 Royce (mood Marguerite neutral (during s2 (effect s2 (spoken Marguerite "Do I hear lord's name in vain" (step s3 (act-type s3 refer (agent s3 Royce (secondary s3 Marguerite (effect s3 (spoken Royce "Good God Mama what is that thing?" (effect s3 (refers Royce Marguerite object (mood Royce angry (during s3 Figure 1 Snapshot of the story world description given as an input to the camera planner. Figure 2 illustrates the abstract operators that we used for filming the sequences. The affective information within the story is used as constraints on the possible selection of primitive shottypes. For instance, the abstract action Film-Dialog is further expanded with a primitive close-up shot instead of a long-shot if the mood of the character in the story is angry through the constraint (mood Marguerite angry (during s3. Further having the temporal index predicates like (during s3 provides the authors with an expressive language for specifically representing story characteristics. Low-angle primitive shots are used to indicate dominance of one character over another in a scene. For a scene where the relationship between the characters is unspecified or if the characters are not closely related, long-shots are preferred over close-ups. In the example shown in Figure 1, close-shots are used as the conversation participants are closely related. A run-time visualization of the plan-space for this example is shown in Figure 3.
(define (decomposition tell-story :parameters (?story :constraints ((story?story (story-conv?story?scenes :links( :steps ( (forall?scene in?scenes (step2 (film-conversation?scene :orderings ( :rewrites (((BEL V (story?story ((forall?scene in?scenes (BEL V (Occurs?scene (define (decomposition film-conversation :parameters (?c :constraints ((conversation?c (conv-steps?c?slist :steps ( (step1 (apex-shot?c (forall?step in?slist (step2 (film-dialog?step :orderings ((step1 step2 :rewrites (((BEL V (Occurs?c ((forall?step in?slist (BEL V (Occurs?step (define (action film-conversation :parameters (?c :precondition NIL :primitive NIL :constraints NIL :effect ((BEL V (Occurs?c Figure 2 Representation of film idioms as plan operators (simplified We implemented a scenario from the movie Rope by Alfred Hitchcock on the Unreal Tournament 2003 game engine. We used the camera planning implementation described in [1] for realization of the story.
Figure 3: Visualization of plan space of the camera planner Conclusions We implemented a camera planning system for automatically generating shots based on the context and characteristics of the underlying narrative. From the analysis of various conversational sequences in movies we found that there were certain geometric blocking formations that were common across different types of conversation sequences. Further we concluded that choice of camera angles was affected by the mood of the character in the shot as well as the relationship between the characters within a conversation. We also applied a more cinematic rendering to the theatrically oriented filming of the movie Rope. The results are presented as screenshots in the Appendix B of this document. References 1. Jhala Arnav, Intelligent Cinematic Camera Planning for Dynamic Narratives, Masters Thesis, North Carolina State University, 2004. 2. Young, R. Michael, Riedl, Mark, Branly, Mark, Jhala, Arnav, Martin, R.J. and Saretto, C.J., An architecture for integrating plan-based behavior generation with interactive game environments, The Journal of Game Development, 1, March 2004. Appendix A Links to screenshots for the movies analyzed. o Midnight in the Garden of Good and Evil, Conversation sequences: http://liquidnarrative.csc.ncsu.edu/~ahjhala/midnightgoodevilpics.html o Rope: http://liquidnarrative.csc.ncsu.edu/~ahjhala/rope.html o A Few Good Men: http://liquidnarratives.csc.ncsu.edu/~ahjhala/fewgoodmen/fewgodmen.html
Appendix B
Shots generated by the system that were not possible given the constraints of the actual filming of the movie.