What is ReelSmart Twixtor?

Transcription

1 1 of 9 4/6/ :27 PM TM TM current version: 1.3 What is ReelSmart Twixtor? ReelSmart Twixtor intelligently slows down or speeds up your image sequences. ReelSmart Twixtor enables you to "time warp" your image sequences with visually stunning results. In order to achieve its unparalleled image quality, Twixtor synthesizes unique new frames by warping and interpolating frames of the original sequence... employing RE:Vision's proprietary tracking technology that calculates motion for each individual pixel. ReelSmart Twixtor AVX plugin works in the following apps: Windows applications: Symphony 2.0 and later, Media Composer 9.0 and later, XPress 3.0 and later, XPress DV 1.5 and later. Macintosh applications: MediaComposer 10.1 and later, XPress 4.0 and later. Twixtor is the solution for many tough problems: 1. Your clip only is 5 seconds but your edit list demands 6 seconds. Let Twixtor create the extra frames you need! 2. You need superior quality slow-motion! 3. You need to deliver a 3D animation render in 2 days. You calculate that the rendering alone could take 2 days and you are not going to make it. Render on two's and let the Twixtor save your life by creating the inbetween frames in a fraction of the time! 4. Your director wants cartoonish or robotic motions using a live-action source. Twixtor allows you to keyframe the retiming so you have complete control of the mapping of frames from the input sequence to the output sequence. Twixtor's remapping is animatable which enables you to speedup and slowdown over a sequence with precise control. Twixtor Features: Calculation of motion vectors at each pixel Warping and interpolation of frames to "time warp" input footage Smart processing of fields in the input and output as appropriate Stretching of a sequence using a single scaling parameter Keyframeable retiming for complete control on a frame by frame basis. Before You Start, EXTREMELY IMPORTANT. Common problems discussed here! For all applications If you never tried Twixtor try setting Twixtor's "Stretch Factor" to 2 and "Vectors" to Medium and go! This will make your sequence twice as long. Twixtor by it's nature modifies the duration of a sequence. The basic question to be answered: Is this making the sequence shorter or longer? When applying Twixtor to a source, if the output duration is less than the source duration, then Twixtor can be applied to the source directly. If the output is longer you will have to extend the duration before by creating a dummy track in your timeline of the correct duration and applying the filter from there by referencing the clip you are interested in. Details are presented below.. Pulldown... if you have 3:2 pulldown in your source footage, you will want to remove 3:2 pulldown before processing with Twixtor. Duplicate fields will cause Twixtor to create incorrect inbetween frames. IN THE FOLLOWING DOCUMENTATION NOTES, GUI ITEMS ARE LABELED IN BLUE. Usage of ReelSmart TM Twixtor Quality: # High or Low: Twixtor will render in draft mode if set to Low. Frame Interpolation: A menu option with 2 choices.

2 2 of 9 4/6/ :27 PM 1. Nearest: if the current output frame falls between two input frame times, Twixtor uses the closest (in time) of the two input frames and warps it appropriately. 2. Blend: Performs bi-directional warping and then blends the resulting two warped images.. The more you stretch a clip, the more this choice is recommended. Motion Vector Quality: A menu option with 5 choices. With "High Quality" set: No Motion Vectors: no motion vectors are calculated Sloppy: motion vectors are calculated at 1/2 resolution, then interpolated. Furthermore, a faster but less accurate method is used. Medium: motion vectors are calculated at 1/2 resolution, then interpolated. Furthermore, a slower but more accurate method is used. High: motion vectors are calculated at full resolution, then interpolated. Furthermore, the faster but less accurate method is used. Best: motion vectors are calculated at full resolution, then interpolated. Furthermore, the slower but more accurate method is used. For "Draft " mode we calculate motion vectors at half the resolution as specified above. For demonstration purposes, let's assume that we are scaling a clip's duration by a factor of 3. As such, output frame 25 would conceptually need input frame How is frame calculated? Case 1: Interpolation: Nearest with Motion Vector Quality: No Motion Vectors. In this case, no warping can occur (there are no motion vectors!), so the closest input frame in time (input frame 8) is returned. This is equivalent to Time Stretching or Time Remapping in After Effects without frame blending. Case 2: Interpolation: Blend with Motion Vector Quality: No Motion Vectors. In this case, no warping can occur (there are no motion vectors!). However, we've asked the module to blend frames, so we'll get a return frame that is a dissolve of frame 8 (at 67%) and frame 9 (33%). This is similar to After Effects with frame blending turned on. Case 3: Interpolation: Nearest with Motion Vector Quality set to anything other than No Motion Vectors. The motion is calculated from input frame 8 to input frame 9. Frame 8 is then warped using the motion calculated (scaled by 1/3) and returned. Case 4: Interpolation: Blend with Motion Vector Quality set to anything other than No Motion Vectors. The motion is calculated from input frame 8 to input frame 9. Frame 8 is then warped using the motion calculated (scaled by 1/3) and returned. Frame 9 is warped using the motion calculated from frame 9 to frame 8 (scaled by 2/3). Finally, these two warped frames are blended as in Case 2. Note that if fewer frames are being generated for the output sequence, that is, the input sequence is being speed up (total time duration is shrunk), then motion blur may then be applied (see Motion Blur option below). Layered Motion: Multiple Layers of Motion? Sometimes the interpolator will perform better on footage that has multiple layers of motion if this box is checked. Note that this option increases motion vector calculation time significantly. Motion Blur: When speeding up a clip, you are essentially telling Twixtor that there is "more action" happening per frame. In order to simimulate what happens in a real camera, and to reduce flickering, Twixtor applies motion blur when speeding up footage in order to simulate what happens when more motion takes place in the same amount of (virtual) shutter opening of a camera. You can dial in how much motion blur is added with the Motion Blur slider. Note this is not the same as our ReelSmart Motion Blur plugin. This is just a shutter compensation simulation. Get Motion From SecondLayer: Allows you to specify another clip as timewarping source. This is one way to make a segment longer then it is. For example put a dummy movie clip of the desired final length on video track2 (on top, and turn that ON so this is what you see in the viewer as output), then put the segment you want to make slower underneath (video track1). Apply the effect to the top segment. Turn on "Get Motion from Second Layer". This will "pull" frames from the bottom track (the track you care about), but output them to the top track. Note for Avid novices that you can import a single frame (like a black jpeg) to make arbitrary duration segments. Stretch: Changing sequence timing. You'll need to decide if you are stretching your input source at a constant rate, that means for example "double duration". If so, make sure Stretching Method is set to Constant Stretch. The other option is Keyframed Stretch, which informs Twixtor you'll be keyframing your time resequencing. If Constant Stretch is chosen, then Twixtor assumes that you'll be changing the duration of the clip by the amount specified in the Constant Stretch Factor parameter. For example, if you want a 1 second sequence to be 2 seconds in length, then

3 3 of 9 4/6/ :27 PM set this value to 2. Note that if you have a 1 second clip that you want to make 2 seconds then you'll have to apply the effect by first creating a dummy track of the final duration (in this case, 2 seconds) and putting it on a different track from the material to be stretched. For example, put a dummy movie clip of the desired final length on video track2 (on top, and turn te track with the dummy movie ON so this is what you see in the viewer as output), then put the segment you want to make slower underneath (video track1). Apply the effect to the top segment. Turn on "Get Motion from Second Layer". This will "pull" frames from the bottom track (the track you care about), but output them to the top track. Note for Avid novices that you can import a single frame (like a black jpeg) to make arbitrary duration segments. Note: the Source Frame (for keyframing) parameter is used when you want to dynamically remap time. If Stretch Method is set to Constant Stretch then the Frame parameter is ignored. Conversely, the scaling slider should not be used when key-framing destination frames. TIP: If you are doing a complex dynamic remapping of the video action, you can quickly set your key-frames by setting Motion Vector Quality to None and Frame Interpolation set to Blend. When done setting keyframes, you can go back and set the Motion Vector Quality to the desired quality. EXERCISE: If you are reading this manual for the first time, you should try taking a movie you have lying around and try to expand or contract it's duration using Constant Stretch. KEY-FRAMING The Time parameter allows you to control the shape of the retiming curve. It is meant to be keyframed where you set "Source Frame" keys. Note this is supplied as Avid does not provide non-linear parameter interpolation and this is essential for nice progressive "slow-downs". In order to set a keyframe, you need to first add one using the "Add Keyframe" button at the bottom of the Effect Editor and then set your value. Then in the Composer you can move between keyframes. After selection of a keyframe, new values can be set for the selected key in the Effect editor.. Note in-between key-frames, the animated values will not be updated on the user interface controls.

4 4 of 9 4/6/ :27 PM Figure showing how to move select a keyframe in the Composer (a triangle correpsponds to a keyframe). After selection of the keyframe, new values can be set for that key in the Effect editor. This section describes keyframing in the context of the remapping of time. If you wish to keyframe the timing then make sure Stretch Method is set to Keyframed Stretch. (Constant Stretch Factor is ignored in this case, and the Frame (for Keyframing) parameter is used instead. This parameter will be referred to as Frame for the rest of this document. The value of the Frame parameter tells Twixtor which input frame should be returned for the current time (fractional values are interpolated as described in the previous section). Note to time remapping novices: Immediately after applying Twixtor to a clip, it is advisable that you: First, decide if you want to make the Frame parameter animatable. Second, if so, then you are advised to make keyframes for the Frame parameter as follows: make a keyframe with value 0 at time 0 make a keyframe at the last time value in the sequence, with a value equal to the last frame number. That is if your output sequence needs to be 1000 frames and your source is 100 frames, key in 100 for Source Frame value at frame Or other example: If you don't want to change duration but only the rate within the clip then let's say you have a sequence of 300 frames. make a keyframe with a value of 0 at time 0 make a keyframe with value 300 at time 300 (time, in this case, is in "frames" instead of timecode). This will give you an identity mapping (frame 0 returned at time 0, frame 300 returned at time 300, frame 47 will be returned at time 47, etc.). Note to Avid keyframing novices: Keyframes as you might know are global (all the values inherit the "Add KeyFrame" mark) in many Avid systems, and as

5 5 of 9 4/6/ :27 PM well the value between two keyframes is interpreted linearly which is often not what you desire if you wish to perform smooth slow-downs and speed-ups. Time KeyFrame Behavior: Options are: Default, Constant, Linear, EaseIn, EaseOut, EaseIn-EaseOut You have to be aware that normally Avid internally only supports linear keyframing and nothing else. Ease-in and Ease-out allow you to smooth the behavior between two keys (making the change in speed smoother across the keyframe). "Constant" keyframe behavior in this context allows you to create a freeze frame. Let's imagine that we have 3 keys, at frame 1 we have a value of 1. At time 10 we have a key with value 10. At time 20 we have a key with value 15. As you can see, Constant and Linear interpolations behave as expected. Ease-in starts out from the key slowly, then accelerates toward the other key. Ease-out does just the opposite, the interpolation starts out fast then slows down. Ease in and out does both. You can mix interpolation styles. The interpolation style for a key is held valid until the next key (see the figures below). Note you cannot see these keyframe values in an Avid System, nor visualize the curves shown below, so it might be a good idea to limit the number of them and keep a note of them somewhere. Constant Interpolation on each key. Linear Interpolation on each key. Ease In Interpolation on each key. Ease Out Interpolation on each key.

6 6 of 9 4/6/ :27 PM Ease In/Out Interpolation set on each key. Linear interpolation set on key at time 1. Ease In/Out interpolation set on key at time 10. Visualation of interpolation styles. A key at time 1 has been given value 1. At time 10, value 10. And a key at time 20 is set to 15. Again, you cannot see these keyframe values in an Avid System, nor visualize the curves shown below, so it might be a good idea to limit the number of them and keep a note of them somewhere. Okay, you ask, what happens when there are fields in the input (interlaced video)? Input and Output Fields # Options: Default, Full Frame, Upper, Lower # Default is what is set for the project. The Default (Auto) is necessarily one of the 3 other options, Full Frame (Progressive) or Lower Field Dominant or Upper Field Dominant. IMPORTANT AVID USERS NOTE: ODD in Avid corresponds to UPPER FIELD DOMINANCE in Twixtor. EVEN in Avid corresponds to LOWER FIELD DOMINANCE in Twixtor. So recap here, imagine you have a frame which has black on the first line and white on the second line, then black on the third line, then white on fourth, etc... That is you have a black field and a white field interlaced into a full frame. Upper field dominance means that the first line will be first in time, and the second line (therefore the second field) will be half a frame (timewise) later. Lower dominance means that the second line is the first in time. Avid nonmenclature might be confusing because they use ODD and EVEN as a means to state what the first or top line is, assuming that the two fields are stored independantly as they are on a videotape frame. Of course if you have progressive frames make sure it is not processing in fields. We supply ways for you to bypass whatever the AVID default is. Of course you are fully aware of these types of problems if you are currently trying to import PAL DV (lower field dominance) in a PAL AVID setup with other video sources for example. Make sure you import your movie with the correct field interpretation otherwise the picture will have a large studder. Inversely it the picture abnormaly shakes, ask yourself if you have inadvertly reversed the dominance. Also, it is not recommanded to use Twixtor to simply reverse dominance, this would be better accomplished by sliding a sequence by a field or moving it spatially by one pixel vertically. Avid Import Panel:

7 7 of 9 4/6/ :27 PM Some things to understand before you think there is a bug... Twixtor allows you to create full frames from fields using field interpolation. This is accomplished by setting "Full Frame" in the Twixtor Field Output setting and then setting the Field Input to "Upper" or "Lower" as appropriate. Internally 30i footage really looks like 60p to Twixtor. A standard algorithm is applied to make each frame (we interpolate the fields to create a whole frame). Input frame 0.0 is made from the dominant field of the first frame. Frame 0.5 is made from the non-dominant field of the first frame and so on. Be aware that Avid still thinks that you are in field mode. In an Avid system when you apply a filter to an image segment, the viewer then advances field by field so it will show two times the same image when you output Frames from fields or, more accurately, the two fields which comprise one whole frame. Also, important note, in this version, when you go from fields to full frame, in effect the stretching is divided by 2 as you are internally going from 60 frames to 30 frames, so you probably want to divide by 2 the desired stretching in such case. Know Issues: It appears when we test this in DV XPress Windows that on Export to a QuickTime movie Avid can trim the first frame out (saves frame 1 as frame 0). PROBLEMS IN INPUT SEQUENCES If you have not used Twixtor yet, this section might be a little esoteric and we suggest you forget it for now. This information will be of more use to you as you become more familiar with Twixtor and want to understand how to generate the best possible results. And as well gives you an idea of the problems we are working on. We are not saying that Twixtor works all the time, we are saying most of the time it works better then frame blending. So if you encounter a bad frame here and there, simply animate the Vectors option menu so that you use No Motion on bad frames. And most of the time that will be an acceptable workaround. Also some of the suggestions here are not necessarily available within some Avid Systems, we nonetheless tell them as many of you have different softwares in-house. It might happen that Twixtor might do poorly in specific instances. The idea here is not to discourage you and tell you it does not work, but the inverse which is truth in advertising. As you become an expert user, you learn to predict the kind of material that can cause problems. 1. "Picket Fence": A very regular pattern in motion (for example someone wearing a t-shirt with fine stripes) with an object moving in front of it (e.g. the same person hand for instance) might confuse the motion vector calculation. 2. "Transparency": Overlay of semi-opaque surfaces might create unexpected results. Some cameras for example will streak under fast motion and that can create disappointing results.

8 8 of 9 4/6/ :27 PM 3. "Short Interval Defects" : Sudden global illumination change (eg flash), strobing, dust,... can create unexpected / undesired results. For example if there is a piece of hair or a scratch on the scanned film for a frame then you should clean such defect before processing. 4. "Fields": We provide some basic field handling. Always remember that by definition, even and odd scan lines in material with fields are considered half a frame apart. It is not correct to process material with fields as if they were frames. 5. "Duplicated Frames": We do not yet provide automatic duplicated frame/fields detection support. You have to be aware that if your material has 3:2 pulldown you should remove it beforehand. It is our understanding that Avid Systems now bundle Commotion. If your applications does not support 3:2 pulldown removal, you can always remove it before hand in Commotion (thus making it shorter, so you will have when twixting your sequence to add that difference 5/4 or if you prefer add a scale of 1.25, meaning if you want to double the duration of a movie that has 3:2 pulldown, you would remove the 3:2 pulldown, then stretch it by 2*1.25 or 2.5. You do not need in our opinion to readd a 3:2 pulldown. The same applies to animations on 2's. 6. "Alternate Motions": When motions going in different direction are layered it is possible that the dominant motion affects the background motion. 7. "Specular Highlights": If you have moving lights, e.g., a shiny object that reflects the light as it moves, this might cause problems because as the motion vector calculation attempts to match two images, it will tend to follow the highlight. 8. "Ultra-Fast Structured Motion": We are very perceptually sensitive to human actions. We have seen sometimes that certain complex rapid motion such as someone doing a frenetic dance create interframe displacements that are just too big for our solver in a manner that is completely satisfying. 9. "Fast Cuts": Although it makes sense why, we nonetheless say it loud here, that if you perform this on some final fast-cut advertising or some MTV like footage that has dissolves and multi-layered of actions going in and out, it will do something but the result might be more predictable if used on the unprocessed source elements individually, and then apply the transitions. 10. "Occlusions": Problems caused by object motions tend to be one frame problems and localized in an area of the frame which is called occlusion, which is some pixels that you see on one frame but are not visible on the other as a result of that object motion. 11. "Compression Artifacts": Certain video coding techniques such as DV compressors use 8 by 8 pixels blocks (intraframe) based compression. What this means for you is that if for instance you have a sharp edge that moves, it will switch of 8 by 8 block and therefore locally it's neighbor values will be all different (substantially not like 2-3 values over 255 but sometimes 40 off). And Avid appears to do this on a field basis. This is one reason why green screen like setups perform badly with DV aside from being 4:1:1... With material such as this, lower accuracy motion estimation might produce more desirable results (unclear). Also with DV-like sources, avoid nearest interpolation mode, the blending mode actually helps by itself to reduce interframes noise. 12. "Very Large Motion" or "Fast Action": As a rule of thumb consider that Twixtor will be most accurate for pixel displacements that are a maximum of 5% of you image resolution (for 720x486, this comes to a maximum displacement of 35 pixels or so). With displacements bigger from that Twixtor will start to become less precise as it tries to separate motions from one another in an image sequence. So, with fast action content, you will probably get better results with Twixtor from material captured with a 30i FPS camera than with a DV camera that does 12 progressive FPS, because the same action will move farther, per frame, in the 12fps footage than in the 30fps footage. If you have the option of shooting progressive or interlaced video, a basic rule of thumb might be if you have fast action (or because you are running with a hand-held camera) shoot interlaced (because we construct 60fps from the 60 fields, which gives us more frames to deal with, which reduces the motion per frame), otherwise shoot progressive (because we don't have to reconstruct a frame from a field, you will get better quality results). AESTHETIC CONSIDERATIONS: Gee! It breaks on these 3 frames what do I do? Now that we have discouraged you, let's help you to be creative in repairing problems. We know that most of the time, there is nothing we can do to prevent problems at the source because the material already exists, and you need to fix it. So let us share a bit our experience or 2 frame defects on source material: Sometimes you receive a movie with one or two frame defects (could be a flash for instance). Take the good frames on both ends of the bad ones and slice them together into another comp, create a short Twixtor clip of the proper duration and reinsert it back over the bad frames. Naturally if you also want to Twixt the whole movie, perform the repair before applying Twixtor to the whole sequence.. 2. Splotches: Although we try to accommodate for separate layers with the Separate Motions button there are times when you will still get some splotches (unwanted warping artifacts). You can probably run Twixtor twice (once with Motion Vector Quality set to Best and once with Motion Vector Quality set to None, for example). If you are scaling by a low-factor (under 1.5) then you might want to swap the frames that are bad. If your problem is very local, you can create a matte to reveal the None version from within the Best version, using the compositing functions of your application. You might want to do a soft matte for nice blending. 3. Ghosts: Since errors such as gosting are usually due to occlusions (pixels that don't have correspondance in the other frames due to motion), they are usually easy to paint out. Some camera motion such as driving on a bridge tend to create a multi-frame ghost which usually is not too problematic to paint out. 4. Using existing mattes: Sometimes, rotoing out a fast moving foreground object will make the background very stable and produce tracking data without the foreground motion influence. If you have the luxury of having mattes for a layer, you might want to Twixt the foreground and background separately (so that the motion tracking influences on the layers go away or are reduced by doing Twixting them separately). 5. Shakes...: Although by definition we deal with camera shakes... note that when you slowmo something you will also slowmo the camera shake and vice-versa, so if possible you might consider to pre-stabilize the sequence on which you want to apply Twixtor. 6. Multiple Segments: When you are applying Twixtor to a sequence that has multiple cuts in it, it might be preferable when doing

9 9 of 9 4/6/ :27 PM large scale slowmotion to separate the shots and apply Twixtor individually so as to preserve a sharp cut. Also if you have dissolves between two such segments, you will have to decide (if it's possible) if you want to do the slow-mo on the result or before, note you can always turn Motion Vector Quality to None for the dissolve segment or inversely you might like the fluid resulting transition. 7. Noise, compression artifacts: Our algorithms should not be too sensitive to small grain noise at the analysis level, however it might look weird to extrapolate a lot of frames per frame on material that contains a lot of noise (as it essentially slow down the noise also) so you are advise to remove some noise, perhaps running a noise-reduction filter before. Perhaps also you should consider that even perfect inbetweening with a computer animation systems can look weird as it won't have any local dynamics, that is it becomes similar to doing global shape key framing, everything will sort of move at the same rate. 8. Softening: Another related thing to watch out is that you will notice that inbetweens will soften the texture details. The thinking is that it's sort of OK as you don't need as much details when you move (you assume you will have some motion blur) and the image where it does not move will remain very soft. If unwanted softening occurs, you can consider little post tricks including slightly sharpening (maybe simply an unsharp filter) and adding a bit of noise Handling Multiple Motions: A rough garbage matte roto that isolates the foreground object might be necessary on the few frames where the problem occasionally occurs. After applying Twixtor to the frames with the garbage matte (on both foreground and background, with inverse mattes of course!), the inbetweened frames can be recomposited back together. Copyright RE:Vision Effects, Inc.