University of California. Santa Cruz. MPEG-2 Transport over ATM Networks. of the requirements for the degree of. Master of Science

University of California Santa Cruz MPEG-2 Transport over ATM Networks A thesis submitted in partial satisfaction of the requirements for the degree of Master of Science in Computer Engineering by Christos Tryfonas September 1996 The thesis of Christos Tryfonas is approved: Anujan Varma J.J. Garcia-Luna-Aceves Subir Varma Dean of Graduate Studies and Research

iii Contents Abstract Acknowledgments xii xiii 1. Introduction 1 2. MPEG Overview 6 2.1 History...................................... 6 2.2 Color Representation............................... 7 2.3 MPEG Coding.................................. 9 2.3.1 Coding Principles............................. 9 2.3.2 Picture types in MPEG......................... 10 2.3.3 Intraframe DCT coding......................... 12 2.3.4 Quantization............................... 12 2.3.5 Entropy Coding............................. 13 2.3.6 Motion-Compensated Interframe Prediction.............. 14 2.4 MPEG-2 Video Features............................. 15 2.4.1 MPEG-2 Proles and Levels....................... 16 2.5 MPEG-2 Systems Layer............................. 17 2.5.1 Transport Stream............................ 21 3. Audiovisual Formats and Requirements 32 3.1 Video Standards................................. 32 3.2 Clock Requirements............................... 33 3.3 Networking Requirements............................ 35 3.4 Perceptual Requirements............................. 37

iv 4. ATM Networks 40 4.1 Classication of Services in ATM networks................... 40 4.2 QoS support and Scheduling Disciplines.................... 43 5. MPEG-2 over ATM 48 5.1 Choice of Adaptation Layer (AAL)....................... 48 5.1.1 Transport over AAL1.......................... 48 5.1.2 Transport over AAL5.......................... 50 5.2 Transport Packet Encapsulation........................ 52 5.3 Service Class Selection.............................. 55 5.4 Clock Synchronization.............................. 59 6. Experimental Results 60 6.1 Simulation Model................................. 60 6.2 Description of Traces............................... 62 6.3 Low-Pass Filter Design.............................. 65 6.4 Simulation Results................................ 76 6.4.1 Experiment 1............................... 76 6.4.2 Experiment 2............................... 80 6.4.3 Experiment 3............................... 84 6.4.4 Experiment 4............................... 89 6.4.5 Experiment 5............................... 96 6.4.6 Experiment 6............................... 103 6.5 Conclusions.................................... 108 7. Conclusions and Future Work 109 References 112

v List of Figures 2.1 Example of inter-dependence among various picture types in a MPEG video sequence...................................... 11 2.2 DCT and zig-zag scan (regular and alternate)................. 13 2.3 Motion Compensation............................... 14 2.4 MPEG encoder (from [71])............................ 15 2.5 Creation of elementary stream from uncompressed data (from [64])..... 19 2.6 Simplied overview of Systems layer (Transport Stream case)......... 20 2.7 Generation of a packetized elementary stream from an elementary stream.. 21 2.8 Transport Stream generation from PES packets (from [64]).......... 23 2.9 Structure of a transport packet.......................... 24 2.10 Piecewise linearity in Transport Rate...................... 27 2.11 Clock recovery using a Phase-Locked Loop (PLL) (from [31])......... 29 3.1 PAL/NTSC demodulator system block (from [3])................ 34 4.1 Token bucket shaper................................ 44 4.2 Abstract view of a scheduler in the SRPS class (from [70]).......... 46 5.1 Structure of AAL5 layer (from [51])....................... 50 5.2 Protocol stack for MPEG-2 over ATM (from [16])............... 52 5.3 PCR packing schemes for N = 2......................... 54 6.1 Network topology used in the simulations.................... 60 6.2 Protocol stack of the simulation model..................... 61 6.3 ON-OFF trac model............................... 62 6.4 Transport Rate of trace A............................. 63 6.5 Transport Rate of trace B............................. 64

vi 6.6 Block diagram of the PLL used in the MPEG-2 decoder............ 65 6.7 Delays experienced by transport packets containing PCRs under the PCRunaware scheme.................................. 67 6.8 Delays experienced by transport packets containing PCRs under the PCRaware scheme.................................... 68 6.9 Decoder frequency using PCR-unaware and several rst-order LPFs with dierent cuto frequencies............................ 68 6.10 Decoder frequency using PCR-unaware and several second-order LPFs with dierent cuto frequencies............................ 69 6.11 Decoder frequency using PCR-unaware and several third-order LPFs with dierent cuto frequencies............................ 70 6.12 PCR-STC dierence under the PCR-unaware scheme - 1st order LPF with 0.01 Hz cuto................................... 70 6.13 PCR-STC dierence under the PCR-aware scheme - 1st order LPF with 1 Hz cuto........................................ 71 6.14 PCR-STC dierence under the PCR-aware scheme - 2nd order LPF with 1 Hz cuto..................................... 71 6.15 Buer occupancy for trace A under PCR-unaware scheme using 2nd order LPF with cuto frequency of 0.1 Hz....................... 73 6.16 PAL color sub-carrier generation frequency under PCR-unaware scheme using 2nd order LPFs................................ 74 6.17 PAL color sub-carrier generation frequency under PCR-aware scheme using 2nd order LPFs.................................. 74 6.18 Rate of change of PAL color sub-carrier generation frequency under PCRunaware scheme using 2nd order LPFs and averaging over 40 seconds.... 75 6.19 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and FIFO scheduling discipline for experiment 1....... 76

vii 6.20 Delays experienced by transport packets containing PCRs under the PCRaware scheme and FIFO scheduling discipline for experiment 1........ 77 6.21 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and FFQ scheduling discipline for experiment 1....... 77 6.22 PAL color sub-carrier generation frequency under PCR-unaware scheme for experiment 1.................................... 78 6.23 PAL color sub-carrier generation frequency under PCR-aware scheme for experiment 1.................................... 79 6.24 Rate of change of PAL color sub-carrier generation frequency under FIFO scheduling discipline for experiment 1 (averaged over 40 secs)......... 79 6.25 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and FIFO scheduling discipline for experiment 2....... 80 6.26 Delays experienced by transport packets containing PCRs under the PCRaware scheme and FIFO scheduling discipline for experiment 2........ 81 6.27 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and Shaped VirtualClock scheduling discipline for experiment 2....................................... 82 6.28 PAL color sub-carrier generation frequency under PCR-unaware scheme for experiment 2.................................... 82 6.29 PAL color sub-carrier generation frequency under PCR-aware scheme for experiment 2.................................... 83 6.30 Rate of change of PAL color sub-carrier generation frequency under FIFO scheduling discipline for experiment 2...................... 83 6.31 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and FIFO scheduling discipline for experiment 3....... 84 6.32 Delays experienced by transport packets containing PCRs under the PCRaware scheme and FIFO scheduling discipline for experiment 3........ 85

viii 6.33 PAL color sub-carrier generation frequency under PCR-unaware scheme for experiment 3.................................... 85 6.34 PAL color sub-carrier generation frequency under PCR-aware scheme for experiment 3.................................... 86 6.35 Rate of change of PAL color sub-carrier generation frequency under PCRunaware for experiment 3 (averaged over 40 secs)................ 86 6.36 Rate of change of PAL color sub-carrier generation frequency under PCRaware for experiment 3 (averaged over 40 secs)................. 87 6.37 Delays experienced by transport packets containing PCRs under the PCRunaware scheme and SCFQ scheduling discipline for experiment 3...... 87 6.38 Delays experienced by transport packets containing PCRs under the PCRaware scheme and SCFQ scheduling discipline for experiment 3........ 88 6.39 CDF for PCR delays under SCFQ for experiment 4.............. 89 6.40 CDF for PCR delays under FFQ for experiment 4............... 90 6.41 PAL color sub-carrier generation frequency under PCR-unaware scheme and using 120 sources for experiment 4........................ 91 6.42 PAL color sub-carrier generation frequency under PCR-aware scheme and using 120 sources for experiment 4........................ 91 6.43 PAL color sub-carrier generation frequency under PCR-unaware scheme and using 150 sources for experiment 4........................ 92 6.44 PAL color sub-carrier generation frequency under PCR-aware scheme and using 150 sources for experiment 4........................ 92 6.45 Rate of change of PAL color sub-carrier generation frequency under PCRunaware scheme and using 120 sources for experiment 4 (averaged over 40 secs). 93 6.46 Rate of change of PAL color sub-carrier generation frequency under PCRaware scheme and using 120 sources for experiment 4 (averaged over 40 secs). 93 6.47 Rate of change of PAL color sub-carrier generation frequency under PCRunaware scheme and using 150 sources for experiment 4 (averaged over 40 secs). 94

ix 6.48 Rate of change of PAL color sub-carrier generation frequency under PCRaware scheme and using 150 sources for experiment 4 (averaged over 40 secs). 94 6.49 PCR delays' autocorrelation for SCFQ under PCR-unaware scheme for experiment 4..................................... 95 6.50 Delays experienced by transport packets containing PCRs under FIFO scheduling discipline and without any dejittering for experiment 5...... 96 6.51 Delays experienced by transport packets containing PCRs under FIFO scheduling discipline and with 5 ms dejittering buer for experiment 5.... 97 6.52 Delays experienced by transport packets containing PCRs under FIFO scheduling discipline and with 10 ms dejittering buer for experiment 5... 98 6.53 PCR-STC dierence without any dejittering buer for experiment 5..... 98 6.54 PCR-STC dierence with 5 ms dejittering buer for experiment 5...... 99 6.55 PCR-STC dierence with 10 ms dejittering buer for experiment 5..... 99 6.56 PAL color sub-carrier generation frequency under PCR-unaware scheme for experiment 5.................................... 100 6.57 Rate of change of PAL color sub-carrier generation frequency under FIFO scheduling discipline for experiment 5 (averaged over 40 secs)......... 100 6.58 Buer occupancy without any dejittering buer for experiment 5....... 101 6.59 Buer occupancy with 5 ms dejittering buer for experiment 5........ 101 6.60 Buer occupancy with 10 ms dejittering buer for experiment 5....... 102 6.61 NTSC color sub-carrier generation frequency for trace B without cross-trac. 103 6.62 Buer occupancy for trace B without cross-trac............... 104 6.63 CDF for transport packets' delays for experiment 6.............. 104 6.64 CDF for PCR delays for experiment 6...................... 105 6.65 NTSC color sub-carrier generation frequency for experiment 6........ 105 6.66 Rate of change of NTSC color sub-carrier generation frequency for experiment 6 (averaged over 40 secs)............................. 106

x 6.67 Buer occupancy under FIFO for experiment 6................. 106 6.68 PCR-STC dierence under FIFO for experiment 6............... 107

xi List of Tables 2.1 Proles and levels supported in MPEG-2 (from [51]).............. 16 2.2 MPEG-2 Main Prole............................... 17 3.1 Comparison of the three basic video standards................. 33 3.2 Specications for the color sub-carrier of dierent video formats (from [3]). 34 3.3 Delay requirements for various audio-visual services.............. 35 3.4 Error rate requirements for various audio-visual services............ 37 3.5 Cell rate requirements for various audio-visual services............ 38 4.1 Bandwidth demands of broadband services................... 41 4.2 Trac parameters................................. 42 4.3 QoS parameters.................................. 42 4.4 Latency, Fairness and Complexity for several scheduling algorithms. L i is the maximum packet size of connection i and L max the maximum packet size among all connections. V is the number of all the connections, F is the frame size and i is the amount of trac in the frame allocated to connection i. L c is the size of the xed packet (cell) in Weighted Round-Robin........ 45 5.1 Maximum packing jitter for dierent transport rates and N = 2....... 53 6.1 Coecients of the various LPFs used in the experiments........... 66

MPEG-2 Transport over ATM Networks Christos Tryfonas abstract In this thesis, we evaluate a number of schemes for transporting MPEG-2 video streams over ATM networks. The schemes studied include packing schemes at the adaptation layer, scheduling disciplines within the ATM switches, and playback schemes at the decoder. We performed extensive simulation experiments in a multi-hop ATM network using constant bit-rate MPEG-2 Transport Streams produced by hardware encoders with varying levels of cross trac. Our results show that the use of a good fair-queueing scheduler in the ATM switches is essential for providing acceptable quality to the end viewer. Besides, we found that not only the probability distribution function of the jitter but also its autocorrelation plays a signicant role in the perceived quality. High autocorrelation of the jitter for the same probability distribution function may degrade the quality to unacceptable levels. We also observed that, although the quality of the reconstructed clock may be poor in the case of excessive jitter, the MPEG-2 system decoder buer may not be aected signicantly, i.e., may not underow or overow. Keywords: Video, MPEG-2, ATM networks, quality-of-service, trac scheduling.

xiii Acknowledgments This is the right place to acknowledge all the people that directly or indirectly contributed in the successful completion of this work. First, I would like to thank my adviser Prof. Anujan Varma for proposing the research topic, providing a continuous support and directing this work. I would like to thank also Dr. Subir Varma for strongly supporting this work and for his invaluable directions. I would like to thank Prof. J.J. Garcia Luna-Aceves for reading the thesis and providing helpful comments. Without Paul Pitcl and Ari Jahanian from LSI Logic Corporation, Fre Jorritsma from Philips Research Laboratories and Bill Helms and Ji Zhang from Divicom Corporation the search for MPEG-2 Transport Stream traces would have been endless. I would also like to thank Dimitrios Stiliadis for the endless discussions on the subject and Lampros Kalampoukas for the constant support and help with the OPNET network simulation tools. Finally, I would like to thank Diamantis Kourkouzelis for his support and encouragement all these years. Last but not least, I would like to thank my family. Their love, trust and support throughout all these years made the impossible possible. I am grateful to you. September 1996

1 1. Introduction Asynchronous Transfer Mode (ATM) is an emerging standard for broadband networks that allows a wide range of trac types ranging from real-time video to best-eort data to be multiplexed in a single physical network. A key benet of ATM technology is its ability to provide quality-of-service (QoS) guarantees to applications. These QoS guarantees are in the form of bounds on end-to-end delay, delay jitter and packet loss rate. Several classes of service have been dened in the context of ATM networks to satisfy the QoS needs of various applications. The Constant Bit-Rate (CBR) and Real-Time Variable Bit-Rate (RT- VBR) service classes provide upper bounds on delay, jitter, and loss rate. These classes are intended for real-time applications that require low delay and jitter. The Non Real-Time Variable Bit-Rate (NRT-VBR) service class is intended for applications where no jitter control is needed, but a delay guarantee is still required. The Available Bit-Rate (ABR) service class is intended for delay-tolerant best-eort applications and uses a rate-based feedback approach to control potential congestion. Finally, the Unspecied Bit-Rate service (UBR) does not oer any service guarantees and thus, has the lowest priority among all the classes. The feasibility of supporting a specied set of QoS requirements is determined by admission-control algorithms. The use of a trac scheduling algorithm is essential in order to provide QoS guarantees in an ATM network. The function of a scheduling algorithm is to select the packet to be transmitted in the next cycle from all the available packets going through the same output link of a switch. A good scheduling discipline should be able to ensure isolation among the ows so that the desired level of service can be guaranteed to real-time ows even in the presence of other, possibly mis-behaving, ows. In addition, the scheduler must also allow best-eort applications to share the available bandwidth. Several scheduling disciplines have been proposed in the literature to achieve this objective. FIFO scheduling discipline, although the simplest to implement, does not provide any isolation among the various ows and therefore cannot oer deterministic bandwidth guarantees. Many algorithms

2 that provide bandwidth guarantees to individual sessions are known in the literature [10, 24, 25, 38, 59, 66, 68, 69, 70, 77]. Many of these algorithms are also capable of providing deterministic delay guarantees when the burstiness of the session trac is bounded (as in the case of the output of a leaky-bucket shaper). This thesis deals with the transport of real-time trac generated by MPEG-2 applications in an ATM network. MPEG-2 is the emerging standard for audio and video compression. Being capable of exploiting both spatial and temporal redundancies, it achieves compression ratios up to 200:1 and can encode a video or audio source to almost any level of quality. MPEG-2 standard oers two ways to multiplex elementary audio, video or private streams to form a program: the MPEG-2 Program Stream and the MPEG-2 Transport Stream formats. The MPEG-2 Transport Stream is the approach suggested for transporting MPEG-2 over noisy environments, such as in an ATM network. Using explicit timestamps (called Program Clock References or PCRs in MPEG-2 terminology), MPEG-2 Transport Streams ensure synchronization and continuity, and provide ways to facilitate the clock recovery at the decoder end. The transport of MPEG-2 over ATM introduces several issues that must be addressed in order to attack the problem on an end-to-end basis. These include the choice of the adaptation layer, method of encapsulation of MPEG-2 packets in AAL packets, choice of scheduling algorithms in the ATM network for control of delay and jitter, and the design of the decoder. The choice of adaptation layer involves a number of tradeos [16]. Use of a circuit-emulation type of adaptation layer (AAL1) would eliminate the various synchronization problems associated with MPEG-2 but can be used only with constant bitrate MPEG-2 Transport Streams. In the more general variable bit-rate (VBR) case, such an adaptation layer cannot be used. An alternative is Adaptation Layer 5 (AAL5) which was initially proposed to carry data trac over ATM networks. Two distinct approaches have been proposed for encapsulation of MPEG-2 streams in AAL5 packets [1]. In the rst approach (PCR-aware), the packetization is done ensuring that when a packet contains a PCR value it will be the last packet encapsulated. This reduces the jitter experienced

3 by PCR values during packetization. In the second approach (PCR-unaware), the sender does not check whether PCR values are contained within a transport packet and may therefore introduce signicant jitter to PCR values during the encapsulation. Finally, a new adaptation layer (AAL2), which is not standardized yet, may oer an alternative in the future for transport of VBR MPEG-2 trac. AAL2 will provide support for clock synchronization and superior error control. Dierent proposals have been made for selecting the type of service under which MPEG- 2 is to be transported over ATM [27, 36, 47, 76]. For constant bit-rate MPEG-2 streams, the CBR class of service is the natural choice. Even in this case, the scheduling algorithm employed by the switches may inuence the overall quality signicantly. For the variable bit-rate case, three main approaches have been proposed. The statistical service with rate renegotiation tries to maximize the multiplexing gain by capturing the VBR nature of MPEG-2 [27, 76]. According to this approach, the eective bandwidth of the source during a specic interval is used in order to allocate resources in the network. If enough resources are not available the quality gets degraded and in that sense, the service is statistical. The rate is renegotiated in the long time scale and the way the renegotiation points are selected depends on the exact algorithm. The second approach, based on a feedback-based available bit-rate service, uses feedback information in order to change the coding rate at the output of the MPEG-2 encoder to suit the available bandwidth [35, 36, 47]. In this approach, the service is considered best eort with some minimum guarantees. In the last approach, which provides statistical service without any guarantees (like the one used in the Internet today), the overall quality relies totally on the load of the network. Thus, no QoS can be guaranteed at all. Synchronization issues may arise while transporting MPEG-2 over ATM due to cell delay variation (jitter). The presence of jitter introduced by the underlying ATM network may distort the reconstructed clock at the MPEG-2 audio/video decoder, which in turn may degrade the quality since the synchronization signals for display of the video frames are obtained from the recovered clock. The common solution is to use a dejittering mechanism

4 at the receiver that absorbs any jitter introduced by the network. In order to ensure acceptable quality at the receiver, each component of the end-to-end path must be designed to provide the desired level of service. Therefore, optimizing only specic components in the path may not be adequate for ensuring the desired quality for the viewer. For example, selecting a low-latency scheduler for the switches while using an adaptation layer that introduces signicant jitter, and a poor phase-locked loop (PLL) design within the MPEG-2 decoder may not be sucient to maintain at the receiver. Therefore, the adaptation layer, encapsulation scheme, scheduling discipline in the ATM switches, dejittering mechanisms at the receiver and the PLL in the MPEG-2 system decoder must all be designed to provide the desired level of quality at the receiver. In this thesis, we evaluate dierent schemes for transporting MPEG-2 Transport Streams over an ATM network. The schemes studied include packing schemes at the adaptation layer, scheduling disciplines in the ATM switches, and playback schemes at the decoder. We performed several experiments in a multi-hop ATM network with varying levels of background trac. We observed that a good fair-queueing scheduler is essential for providing acceptable quality for the viewer. We also found that the perceived quality of the video signal is aected not only by the probability distribution function of the packet delay variation (jitter) but also by its autocorrelation. Finally, we found that, although in specic cases the quality of the reconstructed clock is degraded signicantly, the MPEG-2 system decoder buer may not experience underows or overows. The rest of this thesis is organized as follows: rst, a general overview of the MPEG standard both at the elementary and at the Systems layer will be given in Chapter 2. Then the clock, networking and perceptual requirements for several types of audiovisual applications will be presented in Chapter 3. Aspects dealing with the current types of services in ATM networks and various QoS issues arising from the use of dierent scheduling disciplines in the ATM switches are addressed in Chapter 4. The problem of transporting MPEG-2 over ATM with the approaches that have been proposed in the literature for both the CBR and the VBR cases are discussed in Chapter 5. The various experiments that were performed

5 along with the evaluation of the eciency of dierent schemes for transporting constant bit-rate MPEG-2 Transport Streams over ATM networks are presented in Chapter 6. Finally, Chapter 7 discusses our results, states our conclusions and proposes areas for future research.

6 2. MPEG Overview In this chapter we provide an overview of the MPEG-2 standard. We begin with a short history of MPEG standards and proceed to discuss the basic principles of MPEG coding, such as quantization and interframe prediction. We conclude with a detailed description of the MPEG-2 Systems Layer. 2.1 History Two important standardization eorts were started in the late '80s. One is the ITU-T standard for video-conferencing and video-telephony, known as H.261. The other one came under the name of MPEG (Moving Pictures Experts Group) from ISO/IEC in order to dene a video coding algorithm for application on digital storage media like CD-ROM. In addition, audio coding was added and the scope of the targeted applications was extended to cover almost all applications, from multimedia systems to Video-on-Demand. The activities of JPEG (Joint Photographic Experts Group) [33, 75] played an important role in the denition of MPEG. Although JPEG was intended for still-image compression, it can be also applied for moving pictures, considering the fact that a video sequence is nothing more than a sequence of still images. These still images could be coded individually and displayed sequentially using JPEG but the coded sequence would not take into consideration frameto-frame redundancies that give an additional compression factor. MPEG exploits this temporal redundancy present in any video sequence, in order to maximize the compression ratio. MPEG's rst eort led to the MPEG-1 standard, that was published in 1993 as ISO/IEC 11172. It is divided in three parts: audio compression, video compression and system level multiplexing for applications that need video and audio to be played back in close synchronization. MPEG-1 is being used in a variety of applications. CD-I and Video-CD technology use MPEG-1 as the compression algorithm for video and audio. It was designed

7 to support video coding up to 1.5 Mbps with VHS quality, audio coding at 192 kbps/channel (stereo CD-quality), and is optimized for non-interlaced video signals. MPEG's second eort started in 1990. The main objective was to design a compression standard capable of dierent qualities depending on the bit-rate, from TV broadcast to studio quality. That work led to the MPEG-2 standard which is based on MPEG-1 but is more sophisticated and optimized for interlaced pictures. The MPEG-2 standard is capable of coding standard TV at about 4{9 Mbps to HDTV at 15{25 Mbps. In the audio part of the standard, it supports multi-channel surround sound coding while being backwards compatible with the MPEG-1 Audio denition. Before presenting an overview of the MPEG video compression techniques, a quick overview of the color space will be given. 2.2 Color Representation Color is the perceptual result of light in the visible region of the spectrum, having wavelengths in the region of 400nm to 700nm, incident upon the retina. Physical power (or radiance) is expressed in a spectral power distribution (SPD), often in 31 components spaced in separate 10nm bands. The human retina has three types of color photo-receptor cone cells, which respond to incident radiation with somewhat dierent spectral response curves. A fourth type of photoreceptor cell, the rod, is also present in the retina. Rods are eective only at extremely low light levels (colloquially, night vision), and although important for vision, play no role in image reproduction. Because there are exactly three types of color photo-receptors in the eyes, three numerical components are necessary and sucient to describe a color, providing that appropriate spectral weighting functions are used. This is the concern of the science of colorimetry. In 1931, the Commission Internationale de L'Eclairage (CIE) adopted standard curves for a hypothetical Standard Observer. These curves specify how an SPD can be transformed into a set of three numbers that species a color [9].

8 The CIE system is immediately and almost universally applicable to self-luminous sources and displays. However, the colors produced by reective systems such as photography, printing or paint are a function not only of the colorants but also of the SPD of the ambient illumination. If the application has a strong dependence upon the spectrum of the illuminant, spectral matching must be employed to resolve it. One of the key concepts introduced by the 1931 CIE chart was the isolation of the luminance (or brightness) from the chrominance (or hue). Using the CIE chart as a guideline, the National Television System Committee (NTSC) dened the transmission of signals in a luminance and chrominance format, rather than a format involving the three color components of the television phosphors. The new color space was labeled YIQ, where the letters represent the luminance, in-phase chrominance and quadrature chrominance coordinates, respectively. The PAL and SECAM European television standards are based on an identical color space, the YUV. The only dierence between PAL/SECAM's YUV and NTSC's YIQ color space is a 33 degree rotation in UV. The digital equivalent of YUV is YC b Cr, where the C b chrominance component corresponds to the analog U component and the Cr chrominance component corresponds to the analog V component, with dierent scaling factors. The YC b Cr format concentrates most of the image information into the luminance and less in the chrominance components. The result is that the YC b Cr elements are less correlated and can be coded separately. Another advantage comes from reducing the transmission rates of the C b and Cr chrominance components. The MPEG algorithm strictly species the YC b Cr color space, not YUV or YIQ or any other many ne varieties of color dierence spaces. Regardless of any bit-stream parameters, MPEG-1 and MPEG- 2 Video Main Prole (discussed later) specify the 4:2:0 chroma format, where the color dierence components (C b, Cr) have half the resolution or sample grid density in both the horizontal and vertical direction with respect to luminance.

9 2.3 MPEG Coding 2.3.1 Coding Principles A video sequence has three types of redundancies that a coding scheme needs to exploit in order to achieve very good compression: 1. Spatial 2. Temporal 3. Phychovisual Spatial and temporal redundancies come from the fact that pixel values are not completely independent but are correlated with the values of their neighboring pixels both in space and time (i.e., in this frame and in subsequent or previous frames). This means that their values can be predicted to some extent. On the other hand, phychovisual redundancy has to do with physical limitations of the human eye which has a limited response to nd spatial detail and is less sensitive to detail near edges or shot-changes. Thus, the encoding process may be able to minimize the bit-rate while maintaining constant quality to the human eye at the decoded picture. MPEG uses Discrete Cosine Transform (DCT) and entropy encoding to deal with spatial redundancies (intraframe coding); and motion-compensation and motion-estimation for the temporal redundancies (interframe coding). The specic quantization matrices that are used take into account the phychovisual redundancies. The basic units that the MPEG algorithm uses are: Block: A block is the smallest coding unit in the MPEG algorithm. It is made up of 8 8 pixels and can be one of three types: luminance (Y), red chrominance (Cr) and blue chrominance (C b ). The block is the basic unit in intraframe DCT coded frames. Macroblock: A macroblock is the basic coding unit in the MPEG algorithm. It consists of a 16 16 pixel segment. Since MPEG's video main prole uses the 4:2:0 chroma format, a macroblock consists of four Y, one Cr and one C b blocks. It is the motioncompensation unit.

10 Slice: A slice is horizontal strip within a frame and it is the main processing unit in MPEG. Coding of blocks and macroblocks is feasible only when all the pixels of a slice are available. Besides, coding of a slice is being done independently from its adjacent slices, making it an autonomous unit. Thus, slices serve as resyncronization units. Picture: A picture in MPEG is a single frame in a video sequence. Group-of-Pictures: The Group-of-Pictures (GOP) is simply a small sequence of pictures in which random access is provided. Typical values are 8 and 16 pictures/group. The GOP concept was mandatory in the MPEG-1 standard whereas it is optional in the MPEG-2 standard. Sequence: The sequence consists of a series of pictures (or a series of GOPs if there are). The basic MPEG algorithm consists of the following stages: a motion compensation stage, a transformation stage, a lossy quantization stage and a last lossless coding stage. The motion compensation stage takes the dierence between the current image and a shifted view of the previous one. The transformation stage then tries to concentrate the information energy into the rst transform coecients. The quantization step that follows causes a loss of information that takes into account psychovisual limitations of the human eye, and the last coding stage is nothing more than an entropy coding process that further compresses the data. MPEG is a lossy compression scheme since the reconstructed picture is not identical to the original. If it were lossless then the compression ratio would have been very low (compared to 100:1 that it is typical in MPEG) since the least signicant bits of each color component become progressively more random, thus harder to code. 2.3.2 Picture types in MPEG In MPEG (both MPEG-1 and MPEG-2) there are three types of pictures that are dened [32]: Intraframes or I-frames: These are pictures that are coded autonomously without the need of a reference to another picture. Temporal redundancy is not taken into account.

11 Time Backward prediction Forward Prediction I-frame B-frame P-frame Figure 2.1: Example of inter-dependence among various picture types in a MPEG video sequence. Moderate compression is achieved by exploiting spatial redundancy. An I-frame is always an access point in the video bit-stream. Predictive or P-frames: These frames are coded with respect to a previous I or P-frame using a motion-compensated prediction mechanism. The coding process here exploits both spatial and temporal redundancies. Bidirectionally-predicted or B-frames: The B-frames use both previous and future I or P-frames as a reference for motion-estimation and compensation. They achieve the highest compression ratios. Because they reference both past and future frames the coder has to reorder the pictures that are involved in this process so that each B-frame is produced after all the frames it references. This introduces a reordering delay which depends on the interval between consecutive B-frames. A typical MPEG video sequence is shown in Figure 2.1. The I-frame is coded rst, then the next P-frame and then the interpolated (B-frames) between the two. The process

repeats with the next P-frame and B-frames. 12 2.3.3 Intraframe DCT coding The video energy of the image has low spatial frequency that varies very slowly with time and so a transformation can concentrate the energy in very few coecients. For this transformation, the actual image is divided into blocks to decrease the complexity. Every block (8 8) is transformed according to a two-dimensional Discrete Cosine Transform (DCT) which can be thought as a one dimensional DCT on the columns and a one dimensional DCT on the rows. Each coecient is associated with a specic function of horizontal and vertical frequencies and its value (after the transformation) indicates the contribution of these frequencies in the image block. However, the DCT does not reduce the number of bits required from the block representation. The reduction is being done after the observation that the distribution of coecients is non-uniform. The transformation concentrates as much of the video energy as possible into the low frequencies leading to many coecients being zero or almost zero. The compression is achieved by skipping all those near zero coecients and by quantizing and variable-length coding the remaining ones. 2.3.4 Quantization The quantization stage comes after the DCT transformation stage. The idea here is to transmit the DCT coecients in such a way to minimize the bit-rate. Quantization reduces the number of possible values for the DCT coecients, reducing the required number of bits. This comes from observations showing that numerical precision of the DCT coecients may be reduced without aecting image quality signicantly. The quantization stage takes into consideration the impact of this transform to the human vision. Thus, each coecient is weighted according to its impact on the human eye. Practically, high-frequency coecients are more coarsely quantized than low-frequency ones.

13 2.3.5 Entropy Coding The nal compression stage starts with the serialization of the quantized DCT coef- cients and attempts to exploit any redundancy left. The way the serialization is done aects the nal compression. The DCT coecients are rearranged in a zig-zag manner as it is shown in Figure 2.2. The scanning starts from the coecient with the lowest frequency (DC coecient) and follows the zig-zag pattern until it reaches the last coecient. In MPEG-2 there is an alternate scan pattern that is more ecient with interlaced video signals. The sequence of coecients is then entropy-coded using a variable length code (VLC). Image Samples - 1 Block (8x8) Transform Coefficients DC coef DCT Regular Zig-Zag Scan Alternate Scan Figure 2.2: DCT and zig-zag scan (regular and alternate).

14 The way the VLC allocates code lengths depends on the probability that they are expected to occur. 2.3.6 Motion-Compensated Interframe Prediction I-frame P-frame Best match Search Area Mv Mv, Mh: Motion Vectors Mh Colocated Macroblocks Figure 2.3: Motion Compensation. The interframe prediction is being used in order to exploit temporal redundancies found in the video sequence. It is used in both B- and P-frames. The idea is to check the displacement of the various macroblocks and to encode the best resulting dierence (see gure 2.3). The MPEG syntax species how to represent the motion information: one or two motion vectors per 16 16 sub-block of the picture depending on the type of motion compensation (forward or backward-predicted). However, the method used in computing the motion vectors is not specied. This can be done either exhaustively or using dierent techniques depending on many parameters. For example, in stationary scenes the predictor may use the same block from the reference frame. If the scene is not stationary then one way to compute motion vectors is to nd the dierence between the current block and a

15 Video in I-frame DCT Q VLC coded bitstream - P- or B-frame IQ IDCT DCT: Discrete Cosine Transform Q: Quantization IQ: Inverse Quantization IDCT: Inverse DCT MCP: Motion-Compensated Prediction VLC: Variable Length Coder MCP + Figure 2.4: MPEG encoder (from [71]). block that is shifted appropriately in the reference frame. \Block-matching" techniques are likely to be used for this purpose. The actual ways of computing the motion vectors is left to the implementor. The prediction is based on the encoded pictures and not on the original ones since the whole encoding process is lossy. The whole encoding process is shown gure 2.4 in the block-diagram of a hypothetical MPEG encoder. 2.4 MPEG-2 Video Features The MPEG-2 standard is similar to MPEG-1 but has extensions to cover a wider range of applications. The primary application targeted during the MPEG-2 denition process was the all-digital transmission of broadcast quality video at coded bit-rates between 4 and 9 Mbits/sec. However, the MPEG-2 standard proved to be ecient for other applications also that need higher rates, such as HDTV. The most signicant enhancement over MPEG-1 is the optimization for dealing with interlaced pictures. Some of the important dierences between MPEG-2 and MPEG-1 standards are summarized below:

16 Simple Main SNR scalable Spatially scalable High prole prole prole prole prole High Level no yes no no yes High-1440 Level no yes no yes yes Main Level yes yes yes no yes Low Level no yes yes no no Table 2.1: Proles and levels supported in MPEG-2 (from [51]). 1. MPEG-2 is optimized for interlaced pictures and can represent progressive video sequences also, whereas MPEG-1's syntax is strictly meant for progressive sequences and was optimized for CD-ROM or applications at about 1.5 Mbit/sec. 2. MPEG-2 has more proles and layers (see below). MPEG-2 supports scalable and non-scalable proles. Quality varies from broadcast to studio. 3. Additional Prediction codes for motion-compensation were introduced, as well as more chroma formats. 4. Several other more subtle enhancements (adaptive quantization, 10-bit DCT DC precision, non-linear quantization, VLC tables, improved mismatch control, etc.) were introduced that improved the coding eciency even for progressive pictures. 2.4.1 MPEG-2 Proles and Levels MPEG-2 has dierent proles and levels depending on the targeted application. The proles dene dierent subsets in the MPEG-2 syntax based on the encoding scheme while the levels refer primarily to the resolution of the video signal produced. Initially the standard had three proles (simple, main and next) and four levels (High type 1, High type 2, Main and Low). As an example, the low level refers to a standard image format video with resolution of 352 240 (SIF format). The main level is targeted for CCIR-601 quality whereas the high level is intended for HDTV. More proles were added later. The complete set of proles now consists of the simple, main, SNR scalable, spatially scalable and high

17 Level Max. sampling frames/ Max. Application dimensions second bit-rate Low 352 288 30 4 Mb/s CIF, consumer tape equiv. Main 720 576 30 15 Mb/s CCIR 601, studio TV High-1440 1440 1152 30 63 Mb/s 4 601, consumer HDTV High 1920 1152 30 84 Mb/s production SMPTE 240M std Table 2.2: MPEG-2 Main Prole. proles whereas the levels consist of the high, high-1440, main and low levels. The new scalable proles can work very eciently in a network environment since whenever there is congestion the enhancement layer can be dropped without aecting the basic quality. The most important prole that was rst standardized is the main prole. The importance of this prole, as noted in Table 2.1, has to do with its ability to handle images from the lowest level (MPEG-1 quality) to the highest, making it suitable for HDTV application. Typical bit-rates and targeted applications for the main prole are shown in Table 2.2. 2.5 MPEG-2 Systems Layer The MPEG standard denes a way of multiplexing more than one stream (video or audio) in order to produce a program. A program is considered a single broadcast entity service. For example, \The 11 O'clock news" is considered a program that has individual streams of video, audio and maybe other data such as caption text. The standard denes the way the dierent streams are multiplexed. A program consists of one or more elementary streams. Elementary streams are the basic entities of a program. Typical examples are video or audio. They may or may not be MPEG encoded (MPEG-1 or MPEG-2) since the standard gives the exibility of having streams with private data, i.e., streams whose content is not specied by MPEG. Such streams may carry teletext or other service information that is oered by a specic service provider.

Two schemes are used in the MPEG-2 standard for the multiplexing process. 18 Program Stream: This is analogous and similar to MPEG-1 Systems layer. It is a grouping of video, audio and data elementary streams that have a common time base and are grouped together for delivery in a specic environment. Each Program Stream consists of only one program. The Program Stream is often called Program Stream multiplex. Transport Stream: The Transport Stream combines one or more programs into a single stream. The programs may or may not have a common time base. This type of multiplexing is used in environments where errors are likely and is the default choice for transport over a computer network. The Transport Stream is often called Transport Stream multiplex. Each of the above schemes is optimized for specic environments. The Program Stream is intended for the storage and retrieval of program material from digital storage media. It is intended for use in error-free environments for two reasons. First, it consists of packets that are relatively long (several kilobytes is typical), so the corruption of a packet may lead to the loss of an entire video frame. Second, the packets within the Program Stream context may be of variable length, making it dicult for the decoder to predict the start and nish points of the various packets. The decoder has to rely on the packet-length eld found in the packet header. If this length value is corrupted, loss of synchronization may occur at the decoder end. The DVD standard will make use of the MPEG-2 Program Stream multiplex. The Transport Stream is intended for multi-program applications such as broadcasting and for non error-free environments. A Transport Stream may have one or more programs. The synchronization problem that is obvious in the Program Stream (diculty in detecting the starting and ending bits of a packet in the case of an error) does not exist here since the packets have xed length. Besides, all the packets are given extra error protection using methods such as Reed-Solomon encoding. It is clear that this multiplex is the choice in the case of transporting MPEG-2 over computer networks. However, the Transport Stream is more complex and so more dicult to produce and demultiplex than the Program Stream,

19 Uncompressed Video Stream (CCIR-601)... Frame 1 (830KB) Frame 2 (830KB) Frame 3 (830KB) Frame 4 (830KB) Frame 5 (830KB) MPEG-2 Compression Presentation Unit Access Unit I-frame B-frame B-frame P-frame B-frame Video Elementary Stream... Figure 2.5: Creation of elementary stream from uncompressed data (from [64]). and does not follow the format of the well-established MPEG-1 Systems layer as is case for the latter. We must note here that Program and Transport Streams are designed for dierent applications and their denitions do not follow a layered model. It is possible sometimes to convert from one to the other but one is not a subset or a superset of the other. More specically, we can extract the contents of a program in the Transport Stream format and form a valid Program Stream. Both Program Stream and Transport Stream layers deal with special entities. The whole process starts from the uncompressed data which comes directly from the actual video sequence. Each frame is uncompressed and is called a \presentation unit". The encoder compresses each frame according to the standard and each frame is now called an \access unit". The stream produced by the access units is called \elementary stream" in MPEG terminology. This process is shown in Figure 2.5. Basically, the MPEG encoding transforms the presentation units to access units which now form the video sequence. The same procedure applies to audio also. After the creation of the elementary stream, the next step is its packetization. The

20 MPEG-2 Elementary Encoder Packetizer Video Source Uncompressed Stream MPEG Encoded Stream (elementary stream) Packetized Elementary Stream Systems Layer MUX Transport Stream MPEG-2 Elementary Encoder Packetizer Audio Source Figure 2.6: Simplied overview of Systems layer (Transport Stream case). resulting stream is now called \packetized elementary stream" and the packets are called \PES packets". The way the PES packets are formed is independent from the actual multiplexing procedure (Program or Transport Stream). A simplied overview of this process for the Transport Stream case is shown in Figure 2.6. A PES packet consists of a header and a payload. The payload is nothing more than data bytes taken sequentially from the original elementary stream. There is no specic format for encapsulating data bytes in a PES packet, i.e., there is no requirement to align the start of the access units and the start of the PES packets. This means that an access unit may start at any point within a PES packet as shown in Figure 2.7. In addition, more than one access unit may be present in one PES packet. The way this packetization is done, however, can signicantly aect the nature of the actual packetized stream. For example, if each PES packet contains exactly one video frame (in the case of a video elementary stream), the decoder can determine the start and end of a frame easily. This, however, requires use of variable size packets. On the other hand, if the PES packets are of xed length then the packetization process at the encoder is simpler. The maximum length of a PES packet is xed at 64 Kbytes with the only exception in the case of a video packetized elementary stream when carried over a Transport Stream, where it can be arbitrary. PES packets have special identiers to distinguish themselves from PES packets be-