Implementation of MPEG-2 Trick Modes

Implementation of MPEG-2 Trick Modes Matthew Leditschke and Andrew Johnson Multimedia Services Section Telstra Research Laboratories ABSTRACT: If video on demand services delivered over a broadband network are to offer the same features as their customers have come to expect through use of video tapes, it is important that they offer VCR-like functionality. This paper investigates the issues involved in providing such functionality, in particular visible fast forward and fast reverse. Consideration is given as to why this is a problem when dealing with MPEG-2 coded bitstreams. A number of solutions are presented, with comments made on the varying levels of cost and complexity of each solution, along with the subjective suitability of each. 1. Introduction The MPEG-2 video coding standard [1] will be used as the compression tool for providing a variety of digital video and TV services. One such service is video on demand (VOD), possibly offering a wider range of video material than just movies. An important requirement for a digital VOD service is the ability to provide VCR-like controls, similar to what people are currently used to. Within MPEG-2, such functionality is known as trick modes, and includes the ability to perform visible fast forward and fast reverse. The MPEG-2 standard doesn't say explicitly how trick modes are to be implemented, but it does provide some tools to assist in the task. This leaves some degree of freedom in the design of a VOD system, enabling different trade offs between cost and complexity. The difficulty in implementing fast forward is that it cannot be achieved by simply sending the bits faster. Bandwidth constraints, along with the inability of the decoder to process the bits faster, precludes this. Fast reverse is also likewise restricted, but the bitstream also doesn't make sense to a decoder when played backwards. Thus in a VOD application, the video server must selectively send only some of the bitstream to the decoder. This paper concentrates on how fast forward and fast reverse of MPEG-2 compressed bitstreams stored on a VOD server can be implemented. Consideration will be given to the complexity and subjective quality of each of the techniques presented. 2. User expectations of trick modes People are currently used to the fast forward/reverse provided by VCRs, in particular the picture that is displayed while fast forwarding, providing a visual indication of the current location on the tape. This picture results from the design of VCRs, which use helical scanning. When playing at normal speed, the tape helically scans, reading a picture from every track (as shown by the thin diagonal lines in Figure 1). At 3 times normal speed, for example, the tape is pulled through 3 times faster than it is when playing normally, yet the head spins at the same rate, so only a third of every track is read (the shaded region). This gives a picture consisting of 3 bands, 1 band from each of 3 consecutive pictures. Fast reverse is achieved in a similar manner by pulling the tape through in the other direction. Tape 1 2 3 Picture Figure 1: A simple diagram showing how a helically scanned tape is read at 3 times normal speed, resulting in a picture with 3 separate bands. This fast forward system is a compromise between the need to provide a visual indication of the current location on the tape and the difficulty in reading an entire picture off the tape, when pulling the tape through quickly. This paper concentrates on the implementation of fast forward from an MPEG bitstream stored on a VOD server. Reading an MPEG bitstream from tape is considered in [2], [3] & [4]. Achieving a fast forward system using an MPEG compressed bitstream read from a video server, which looks similar to the fast forward of tape based analogue VCRs, is challenging. This is primarily because of the way in which MPEG achieves its greatest level of compression temporal prediction. This restricts the ability to perform reverse play, restricts which pictures can be selectively played, and limits the ability to use horizontal stripes or bands of a picture. Playing selected parts of the bitstream is further complicated because the different coding modes achieve different levels of compression. For example, if every 1 2 3

12th frame in a sequence is intra coded, these intra coded pictures comprise much more than a 12th of the bitstream. To illustrate this point, Table 1 shows the approximate ratio between the size of I, P and B pictures 1, with every 12th picture intra coded, and every 3rd picture in between coded as a P picture. Sequence Rate (Mbits/sec) Ratio (I:B:P) Basketball 4 4.5 : 2.5 : 1.0 6 3.8 : 2.3 : 1.0 Calendar 4 7.3 : 3.2 : 1.0 6 5.8 : 2.9 : 1.0 Flowers 4 6.7 : 3.0 : 1.0 6 5.3 : 2.6 : 1.0 Table 1: The ratio of the sizes of I, P and B pictures, for difference sequences and bit rates. While it is difficult to achieve VCR-like control of MPEG coded bitstreams on a video server, different interfaces to fast forward/reverse are feasible, because of the ease with which random access is possible. This can be used to provide larger scale positioning control within a movie, as contrasted with the small scale control offered by visible fast forward/reverse. Examples of interfaces exploiting this include: Provide a slider bar across the bottom of the picture which shows the current position within the length of the movie. Adjusting the slider bar adjusts the current position in the movie. Use an auxiliary file which maintains the location of particular scenes in the movie, which can then be quickly accessed. While interfaces such as these replace some of the uses of the VCR-like fast forward and fast reverse capability, it is important that video on demand services also provide VCR-like controls, as people are used to this style of video control. Thus the remainder of this paper will concentrate on how VCR-like control of MPEG coded bitstreams can be implemented. 3. The MPEG standard & trick modes While it has been said that the MPEG-2 standard doesn't define a specific way in which fast forward/reverse is to be implemented, there are a number of aspects of the standard which provide support for trick modes in general. 1 I, P and B pictures are the different ways in which a picture can be coded by an MPEG-2 coder. I pictures are intra coded, without reference to any other pictures. P pictures are coded using prediction from a previous I or P picture. B pictures are coded using bi-directional prediction, from past and/or future I or P pictures. The "Systems" part of the MPEG-2 standard (part 1 of [1]) performs multiplexing and synchronisation of multiple video and audio bitstreams. Before multiplexing, elementary video streams are placed into packetised elementary stream (PES) packets. The header of PES packets contains timing information, along with a trick mode indication flag and control field. The operation of the decoder changes when this flag is encountered: The non-normal speed of decoding and display causes some fields in the video bitstream, such as the vbv_delay and the bit_rate fields, to be incorrect. Once a picture has been decoded, it is to be repeatedly displayed until the next picture has been completely decoded. Gaps in slices 2 are to be filled in with the content of the previous frame, in the same location. The trick mode control field may indicate that only a reduced number of coefficients may have been used in the bitstream. Using this will result in techniques similar to the transcoding techniques discussed in this paper. The MPEG-2 standard also provides a flag in the slice header, which indicates that all of the macroblocks in a slice are intra coded. While this flag is not used in a decoder when it is playing normally, it is included to provide fast forward/reverse. The standard states that it is not a normative requirement for an MPEG-2 decoder to handle this trick mode control field. Thus, an MPEG-2 compliant decoder need not support trick modes. The standard goes on to say, however, that if the trick mode control field is supported, then it must be implemented in the manner specified in the standard. The summary of the techniques presented in this paper, given in Table 4, will indicate which techniques use the trick mode control field. As MPEG-1 doesn't have the trick mode control field, the techniques presented in this paper which don't rely on a decoder implementing this can also be used in conjunction with MPEG-1 decoders. 4. VOD implementation issues Figure 2 shows the key elements in a video on demand system. When implementing such a system, each 2 Slices are an MPEG construct, grouping together a number of adjacent macroblocks having the same vertical position.

component affects the overall cost and complexity of the system. STU Fixed frame rate Fixed processing Network Fixed bandwidth Server Switching Processing Disks Storage Figure 2: The key elements in a video on demand system, showing the implementation issues involved with each element. Set top unit (STU): There is an upper limit on how fast the STU can display pictures, along with how fast the bitstream can be decoded and the amount of memory the STU has. Network: A fixed bandwidth is available, so the bit rate of a fast forward stream must not exceed the bit rate of the normal play bitstream. Server: The server performs two functions, switching data between the disk drives and the transmission network, and possibly performing some processing of the bitstreams before transmission. Disks: Store the compressed bitstreams. The different implementations of fast forward/reverse place different demands on these components. Other issues which arise when implementing fast forward/reverse include the variety of speeds offered, the ability to do reverse, and the way in which the video is loaded onto the server. In particular, some techniques rely on the bitstream being generated in a particular manner, which may require transcoding of precoded bitstreams. 5. Implementation techniques As mentioned in the introduction, implementing fast forward requires the server to select portions of the bitstream to send to the STU. This can either be performed in real time, or off line: Real time: Select which parts of the single bitstream stored on the disks to transmit when the STU requests fast forward information. This uses the processing capability of the server. Off line: Select the parts of the bitstream which will be used to make the fast forward bitstream only once, when the bitstream is being loaded onto the server. The selected portions of the bitstream are stored in a separate file on the server, requiring extra storage space. The amount of the bitstream selected can also vary: Slices: A number of slices from a picture can be selected. Pictures: A number of pictures from a bitstream can be selected. Table 2 summarises the nature of the techniques presented in this paper, indicating when the bitstream is performed, and what parts of the bitstream are selected. Real time Off line Slices Pictures Intra slices ü ü Selective GOP ü ü Selective pictures ü ü Multiple bitstreams ü ü ü Table 2: A summary of the fast forward/reverse techniques presented in this paper. Each of the techniques presented in Table 2 are now discussed in more detail. 5.1 Intra slices When selecting slices from a bitstream to form a fast forward bitstream, the slices chosen must contain only intra coded macroblocks, as the correct prediction information is not in place for non-intra coded macroblocks to be decoded properly. An advantage of only selecting intra slices is that the same bitstream can also be used to provide fast reverse. At a given fast forward speed, there are restrictions as to how many intra slices can be selected, based on the bandwidth restriction. If, for example, 1/N of the slices are intra coded, selecting the intra slices does not produce a speed up of N, because the intra slices comprise much more than 1/N of the bitstream. Table 1 gives an indication of the relative sizes of intra and predictively coded information. There are two ways in which this bandwidth restriction can be dealt with: 1. Recode the intra slices to reduce their bit rate. This adds computational demands, and reduces the quality of the resulting picture, but can produce a visual effect similar to that of VCRs. 2. Don't send every slice. The slices that aren't sent are replaced by information from previous pictures, resulting in slices not being displayed in increasing time order. The operation of this fast forward technique depends on the insertion and placement of intra slices when the

bitstream is generated. This affects the size and number of the bands which appear when fast forwarding, and the appearance of these bands at different speeds. Thus when a coded bitstream is to be loaded onto the server which doesn't contain intra slices, it must be recoded in order to enable this style of fast forward. Another issue to be considered with intra slices is the frequency of intra pictures. If intra slices are added without decreasing the frequency of intra pictures, the coding efficiency will drop, reducing the quality of the normal play bitstream. On the other hand, a reduced frequency of intra pictures will reduce the number of frames available for random access. The visual effect of intra slices depends on the way in which it is implemented. Not sending every slice causes the missing slices to be replaced with the contents of previous pictures. The resulting picture will appear quite disjointed, especially for high motion content. Recoding can eliminate the disjointed bands in the picture, at the cost of a lower picture quality. Both of these techniques work best for a specific fast forward or reverse speed, with increasing levels of disjointedness at other speeds. 5.2 Selective GOP playing When selecting pictures to generate a fast forward bitstream, predictive coded pictures can be selected, provided that the required prediction pictures are also selected. The simplest way in which this can be done is to select groups of pictures 3 (GOP). Pictures within a GOP can be decoded because the first picture in a GOP, in bitstream order, is intra coded. Depending on the picture coding types used, it is possible that not all of the pictures in a selected GOP can be decoded, because some B pictures may refer to pictures outside of the GOP. Figure 3 provides an example of this, showing that the first B picture in a GOP (in display order) couldn't be decoded if previous GOPs were discarded.... P B * I B P... P B * I B P... P B * I B P... GOP GOP GOP Figure 3: An example arrangement of picture coding types, in display order. The B pictures marked with an asterisk can predict from pictures outside of the GOP. 3 A GOP is an optional MPEG-2 construct, denoted by a small header field, which groups together a number of pictures. The first picture in a GOP, in bitstream order, is intra coded. It is intended primarily for random access. Despite this, selecting GOP playing provides quite a simple fast forward implementation, having the following advantages: The average bit rate for a GOP tends to be similar to the average sequence bit rate, so the fast forward bitstream has the same bit rate as the normal play bitstream. A number of fast forward speeds are easily obtained by skipping over more GOPs between the selected GOPs. A disadvantage is that, depending on the types of pictures in a GOP, this technique may be unable to provide fast reverse. P pictures, which use forward prediction, will prohibit the pictures in a GOP from being played in reverse order. The big drawback with selective GOP playing is that it looks like a video playing normally with lots of jumps. As a result it is difficult to realise that the video is being fast forwarded, and different fast forward speeds look the same. This scheme highlights the fact that technical simplicity should be weighed against visual quality. 5.3 Selective picture playing With fast forward at the picture level, an N times speed up is obtained by displaying every N th picture. This is not possible when reading from a bitstream because arbitrary frames cannot be selected, since P and B pictures can only be decoded when the pictures from which they predict have also been decoded. As a result, fast forward bitstreams can be obtained by discarding B pictures, followed by P pictures as the fast forward speed increases. This leaves a bitstream consisting predominantly of I pictures, the bit rate of which is greater than the average sequence bit rate, as shown by the relative picture sizes in Table 1, along with the following example: Assume that a bitstream has been coded with an I picture every 12th picture, and every 3rd picture in between is coded as a P picture. Suppose also that the average size of an I, P and B picture is 400, 200 and 100 Kbits respectively, typical figures when coding at 4 Mbits/second. This gives 12 pictures in a GOP (1 I, 3 P, 8 B) with an average of 1800 Kbits in a GOP. To play through this bitstream at 3 times normal speed with the same bit rate, the average size of a GOP drops to 600 Kbits because the GOP rate has tripled. This only allows 1 I and 1 P picture to be displayed per GOP. Playing at 3 times normal

speed, 4 pictures in 12 are to be displayed, so each picture needs to be displayed twice. While this example shows what is involved with a three fold speed increase, it is also indicative of what happens at other speeds. Selecting pictures in this manner also enables fast reverse to be implemented, with the limitation that only intra pictures can be selected. When looking at this fast forward technique, it is quite noticeable that pictures are repeated, and this is an annoying artefact. Not only are some pictures repeated, different pictures are repeated a differing number of times, further detracting from the appearance of this technique. 5.4 Off line (multiple bitstreams) As mentioned in the introduction to Section 5, the of which parts of the bitstream to use for the fast forward bitstream can be performed only once when the bitstream is loaded onto the server. This produces a secondary bitstream which is stored on the server and used when required, but increases the amount of disk space used on the server. When playing normally from a bitstream, the server needs to keep track of the corresponding location in the secondary bitstream, to minimise the delay in sending the fast forward bitstream to the decoder. The advantages in performing this only once are that: There is more control over the appearance of the fast forward sequence. Because there is not a real-time constraint, transcoding is possible, enabling the fast forward bitstream to be coded at a bit rate lower than that used to code the normal play bitstream. When generating the secondary bitstream by transcoding, it is also possible to reduce the spatial resolution of the images in the sequence. The reduced spatial quality is suitable for fast forward, and further reduces the size of the secondary bitstream. Possible ways in which the secondary bitstream can be generated include, but are not limited to: 1. Take every N th picture from the normal play bitstream and reduce the spatial resolution by a factor of 2 in each dimension. Table 3 shows examples of the size of the secondary bitstream using this approach, as a percentage of the normal play bitstream (coded at 6 Mbits/sec), for different values of N and for different bit rates. N Bit rate (Mbit/sec) Size 2 1 8 % 2 1.5 13 % 3 1 6 % 3 1.5 8 % Table 3: The size of the secondary bitstreams, relative to the normal play bitstream, for different speed ups and bit rates. 2. A fast forward system which looks similar to that of a VCR can be achieved by taking bands, 1/N the size of a picture from every picture in the normal play sequence, then coding the resulting sequence. It is important to note that the generation of these secondary bitstreams depends on how the video being loaded onto the server is received. If an MPEG-2 coded bitstream is received, it first needs to be decoded, and then recoded once parts of the sequence have been selected. If a video tape (such as Betacam or D1) is received, the normal play bitstream is generated by an MPEG-2 coder attached to the server. The secondary bitstream can be generated by either changing the video sequence before coding, or by recoding the normal play bitstream. A secondary bitstream, once it has been generated, offers a single speed of fast forward operation. Faster speeds can be obtained by either producing additional bitstreams, or by using real time from the secondary bitstream (as discussed in the previous sections). If the secondary bitstream is coded at a rate much less than the maximum rate available, the of intra pictures from the secondary bitstream is not as limited by the increase in bit rate that results. The type of pictures used when generating the secondary bitstream determine how well fast reverse can be performed. Coding only with I pictures generates a bitstream which is easily reversed, but is coded inefficiently. Coding with I and B pictures, along with a small amount of reordering, also produces a bitstream that supports fast reverse. 6. Conclusions Table 4 presents a tabular summary of the fast forward/rewind techniques presented in this paper. Intra slices, intra slices with recoding to reduce the bit rate of the intra slices, selective playing of GOPs, selective playing of pictures and multiple bitstreams are considered. Because a lot of these techniques have some flexibility in how they are implemented, multiple responses (separated by slashes) appear in some boxes.

The issues and questions considered when making this comparison include: Is additional disk space, in excess to that required for the normal play bitstream, required? Is specific coding of a bitstream stored on the server, either the normal play or secondary bitstream, required? Is the quality of the normal play bitstream affected by this specific coding? What degree of on-line processing is required by the server to provide fast forward? Are multiple fast forward speeds provided? Is fast reverse possible? Does this technique rely on a decoder which handles the trick mode control field? Is the fast forward bitstream restricted because of the bandwidth limitation? And finally a comment on the subjective suitability of each technique. Additional disk space required? Specific coding required? Affect normal play bitstream? On line processing required? Multiple speeds provided? Fast reverse Intra slices I slices & recode Selective GOPs Selective pictures Multiple bitstreams no no no no yes yes yes no no yes yes yes no no no detailed select & process simple moderate limited yes yes yes tracking / limited / yes provided? yes yes no yes yes Require trickmode yes yes no yes no yes / decoder? FF bandwidth limitation? yes no no yes no Subjective poor / average / suitability? no VCR-like no average good Table 4: A summary of the different fast forward techniques, indicating the relative merits of each. The subjective quality of a technique, as well as the ease and cost of implementation, must be taken into account when comparing different fast forward and fast reverse implementations. Selective GOP playing highlights this, in that it is simple to implement, but provides a poor quality visual fast forward. A comparison between the different techniques also requires a measure of the relative cost of disk space verses processing power on the server. While the additional disk space required for the multiple bitstream technique can be easily estimated, it is more difficult to determine the additional processing power required for the techniques which perform on-line processing. The dimensioning of a video server also affects the decision, with disk space proportional to the number of movies on offer, and processing (along with switching) capacity proportional to the number of users supported. With fast forward/reverse techniques that use on-line processing, the demand on this processing capacity can vary. This complicates the dimensioning problem which ensures that there will always be some processing power available when a user requires it. A user's perception of the service is important. Multiple bitstreams enable a greater level of control of the appearance of the fast forward picture, and also makes for easier dimensioning of a video server. Thus, despite the additional storage overhead and issues related to achieving multiple speeds, multiple bitstreams provides the best way to provide fast forward/reverse functionality for a video on demand system. Acknowledgments The permission of the Director of Telstra Research Laboratories to publish this material is hereby acknowledged. The authors also thank Trevor Long for valuable discussions on VCR technologies. References [1] ISO/IEC, Information Technology - Generic coding of moving pictures and associated audio. IS-13818, November 1994. [2] E. D. Frimout, J. Biemond, R. L. Lagendijk, Trick mode solutions for MPEG tape recording, Proceedings of the SPIE, Volume 2308, pp 311-321, September 1994. [3] E. D. Frimout, J. Biemond, R. L. Lagendijk, Extraction of a dedicated fast playback MPEG bit stream, Proceedings of the SPIE, Volume 2501, pp 76-87, May 1995. [4] F. Lane, J. Boyce, Fast Forward/Fast Reverse for a Digital HDTV VCR, International Workshop on HDTV '93, November 1993, Ottawa Canada.