To my parents. ii

Transcription

1 RATE CONTROL FOR VIDEO TRANSMISSION OVER VARIABLE RATE CHANNELS by Chi-Yuan Hsu A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulællment ofthe Requirements for the Degree DOCTOR OF PHILOSOPHY èelectrical Engineeringè August 1998

2 To my parents. ii

3 Acknowledgements First of all, I would like to thank my advisor Professor Antonio Ortega for his continuous support and help in my research, I also like to thank him for his eæort in creating an enjoyable research environment in our group. I thank all my colleagues in Prof Ortega's research group, for sharing our ideas í and pizza í in every seminar, also for the pleasant collaboration experience and all the fun we have together. I also like to thank all my friends in Signal and Image Processing and University of Southern California for I have avery pleasant stay in Los Angeles for the past years. Special thanks goes to Dr. Amy Reibman of AT&T Bell Laboratories for her insightful suggestion in this research, and generous contribution to the publication of our work in the IEEE Journal on Selected Areas in Communications, for which Dr. Antonio Ortega, Dr. Amy Reibman and I were selected as the winners for the IEEE Communications Society Leonard G. Abraham Prize Paper Award of I also like to thank Dr. Masoud Khansari of Hewlett-Packard Laboratories for the joint collaboration in this research, which made possible the publication of many papers and this dissertation. Iwould like to thank Professor John Silvester, Professor C.-C. Jay Kuo, Professor Ulrich Neumann, Professor Keith Chugg for their time and precious suggestions as being my guidance committee of the dissertation. I also thank Prof. Robert Scholtz for giving me the opportunity to participate his research, which beneæts me a lot in my research. iii

4 I like to thank my parents and my family for their love and endless support. Without their unselæsh support, all of this work would not be possible. iv

5 Contents ii Acknowledgements List Of Tables List Of Figures Abstract iii vii viii xi 1 Introduction Digital Video and Video Transmission Video Compression Variable quantization Other encoder functionalities for video transmission Rate Control for Video Transmission Transmission Channels Video transmission over broadband networks Video transmission over burst-error channels Overview and Contribution Video Transmission Overview of Digital Video Applications Delay Latency and Delay Constraint in Video Transmission Encoding Rate Constraints Related to Delay Constraint Physical Buæer Constraint Constraints on the encoder buæer state Constraints on the allowable channel rate Physical buæer size selection Summary v

6 3 Joint Selection of Source and Channel Rate for VBR Video Transmission Introduction Optimal Rate Control Optimal Encoder and Channel Rate Allocations Optimal rate control for CBR transmission Optimal rate control for VBR transmission Optimization algorithm Simulation and Experimental Results Comparison of CBR and VBR transmission Double leaky bucket policing function in VBR cannel MPEG Video Experiments Conclusions Video Transmission with Real-Time Encoding and Decoding Introduction Delay Constraints Encoding Rate Constraints Formulation of Optimal Rate Control Rate Allocation by Dynamic Programming Rate Allocation by Lagrangian Optimization Conclusions Rate Control for Video Transmission over Burst Error Channel Introduction Channel Error Control Rate Control Approaches Probabilistic Modeling of Channel Behavior Physical Channel Layer Channel models Channel Rate Estimation Expected Distortion Encoding Rate Selection under Estimated Rate Constraints Dynamic programming Lagrangian optimization Encoding Rate Selection for Minimum Expected Distortion Experimental Results and Conclusions Conclusions and Extensions Summary of the Research Future Extension vi

7 List Of Tables 4.1 Summary of notations Transitional probability for the downlink and uplink channels Summary of the characteristics of the channels used in our experiments102 vii

8 List Of Figures 1.1 Structure of a generic video communication system Block diagram of a general DCT-based encoder Video frames that are encoded in diæerent picture types in a predictive video coding scheme Diæerent types of video applications Delay components of a video communication system Timing diagram of encoder and decoder, note that the clocks at encoder and decoder are shifted by æt c the transmission delay Buæer constrained optimization in the CBR channel case Decoder buæer in the receiver end Buæer and leaky bucket constrained optimization in the VBR channel case Encoder buæer fullness for CBR channel. Delay = 10 GOBs Encoder buæer and leaky bucket fullness for VBR channel. Delay = 10 GOBs, Leaky Bucket Size =10æC=52,000 ç bits. Note that using variable channel rates allows us to increase the eæective buæer size when needed Average PSNR of video coded with diæerent delay and leaky bucket constraints Contour of the average PSNR Trace of the encoded bits for each GOB èupperè and frame èbottomè. The encoding rate and channel rate are jointly selected by the proposed algorithm based on DP Trace of PSNR for each GOB èupperè and frame èbottomè. The encoding rate and channel rate are jointly selected by the proposed algorithm based on DP Transmitted bits for each GOB and frame interval when source rate and channel rate are jointly selected in the VBR channel with leaky bucket constraints Best average PSNR of the video sequence for the double leaky bucket case and the other single leaky bucket cases viii

9 3.12 Encoder buæer and bucket fullness of larger bucket in DLB case. Delay = 10 GOBs. Larger bucket: size = 52,000 bits, drain rate = 5,200 bitsègob. Smaller bucket: size = 10,200 bits, drain rate = 5,600 bitsègob Encoder buæer and leaky bucket fullness for VBR channel. Target average rate ç C = 3,400 bitsègob. Delay = 10 GOBs, Leaky Bucket Size = 10 æ ç C =34,000 bitsègob Trace of the encoded bits for each frame and group of picture ègopè using MPEG encoder in the VBR channel. Target average rate ç C = 3,400 bitsègob. Delay = 10 GOBs, Leaky Bucket Size = 10 æ ç C =34,000 bitsègob Trace of PSNR for each frame and group of picture ègopè using MPEG encoder in the VBR channel. Target average rate ç C = 3,400 bitsègob. Delay = 10 GOBs, Leaky Bucket Size = 10 æ ç C =34,000 bitsègob Contour of the average PSNR. Target average rate ç C=5,200 bitsègob Delay constraint intheframe-based encoding scheme Delay constraint intheslice-based encoding scheme Encoding rate constraints for video slices in the encoder buæer at time t System block diagram Trellis tree in dynamic programming for searching the optimal encoding rate allocation Diagram of buæers in the system Two state Markov channel model N-state Markov channel model N-state Markov channel model: Resulting PSNR of the decoded video by DP-Est, LAG-Est, DP-Min algorithms under end-to-end delay constraint from 50 msec to 400 msec. The results ofdp-adv and DP-No Feedback algorithms are also shown for benchmarking comparison N-state Markov channel model: Resulting packet loss rate by DP- Est, LAG-Est, DP-Min algorithms under end-to-end delay constraint from 50 msec to 400 msec. The results of DP-Adv and DP- No Feedback algorithms are also shown for benchmarking comparison Two-state Markov channel model: Resulting PSNR of the decoded video by DP-Est, LAG-Est, DP-Min algorithms under end-to-end delay constraint from 50 msec to 400 msec. The results of DP-Adv and DP-No Feedback algorithms are also shown for benchmarking comparison ix

10 5.7 Two-state Markov channel model: Resulting packet loss rate by DP- Est, LAG-Est, DP-Min algorithms under end-to-end delay constraint from 50 msec to 400 msec. The results of DP-Adv and DP- No Feedback algorithms are also shown for benchmarking comparison.106 x

11 Abstract Video has become a component of modern communications and multimedia applications. Communication networks and channels with very diæerent characteristics, such as circuit-switched networks, packet-switched networks, Internet, ATM-based B-ISDN, and wireless channels, have been explored as the platform for video transmission. However, the underlying networks and channels place diæerent constraints on the transport of video data. To comply with these constraints, it is necessary to implement rate control at the video encoder in a video transmission applications. In this research, the problem of video rate control is studied by considering the constraining factors in a video transmission system èincluding the applicable endto-end delay, channel throughput, and possible transmission errorsè and translating these constraints into the encoding rate. The encoding rate constraints have to be observed by the encoder in order to guarantee the successful decoding and displaying of the video data that are streaming into the decoder. Based on this formulation of the rate control problem, a number of rate control algorithms, aim to maximize the video quality that are transmitted to the decoder, are proposed for video transmission over various types of transmission channels. The ærst communication channel under study for video transmission is a Variable Bit Rate èvbrè network with usage parameter control. A rate control approach is proposed to jointly select the source and channel rates in such VBR transmission xi

12 environment. Another transmission environment under study for video transmission is burst-error channels. An integrated rate control and error control scheme is proposed such that the source encoding rate can be adaptive to the current channel condition. Overall performance of the video transmission, in terms of reconstructed video quality and data loss rate, can be improved. xii

13 Chapter 1 Introduction 1.1 Digital Video and Video Transmission Recent advances in digital video technology have fundamentally changed the way visual information is processed, stored and transfered. As other technological components such as semiconductor, microprocessors, digital communication and digital signal processing become mature and available, digital video is replacing the traditional analog video with the emergence of various applications such as High Definition TV èhdtvè ë1ë, Video CD èvcdè, Digital Versatile Disk èdvdè, Digital Video Broadcasting èdvbè ë2ë, and video conferencing. An advantageous property of digital video is that many digital signal processing techniques èe.g., compression, encoding, encryption and error correctionè can be directly applied on the video signal which has been digitized. Another reason for the trend of such transition from analog to digital in video technology can be attributed to the wide availability of digital storage devices, computers, digital signal processors and digital communication systems, so that video information is more likely to be processed, stored, and transfered on those digital platforms. 1

14 However, video is also one of the most bandwidth-consuming types of information. Even as new technology causes communication channels with ever-increasing bandwidth, compression of the video data is still necessary to achieve more eæcient bandwidth utilization. For example, in the terrestrial TV broadcasting, satellite TV and cable TV applications, more high resolution video channels can be delivered within the assigned bandwidth if each video signal is compressed for transmission. In the applications of video transmission over digital data network, compression of the video data is necessary to reduce the impact and burden on the network traæc. Therefore, the development of eæcient video codecs has attracted substantial research interest. The demand for a common video compression scheme that makes it possible to exchange video information in various applications also triggered several standardization eæorts by organizations such as International Standards Organization èisoè and International Telecommunication UnioníTelecommunication Standardization Sector èitu-tè to develop video compression standards such as MPEG- 1 ë3ë, MPEG-2 ë4ë, H.261 ë5ë and H.263 ë6ë. The MPEG-1 standard addresses the requirement of video data compression up to about 1.5MBitèsec, and targets the application of video storage on media such as CD-ROM, as well as transmission over narrowband communication channel such as Narrowband Integrated Service Digital Network èn-isdnè. The MPEG-2 standard oæers more features and higher video quality than MPEG-1, and aims to provide a generic compression scheme for a vast variety of applications such as digital video broadcasting, DVD and video over broadband ISDN èb-isdnè. H.261 and H.263 standards are developed by ITU-T and are speciæcally designed for real-time low-bit rate video conferencing services over N- ISDN or Public Switched Telephone Network èpstnè where bandwidth constraint and low delay latency are of primary concern. 2

15 While in some cases digital video is stored locally, inanumber of video applications a communication channel is used to transfer and distribute the video information. This form of video communication applications includes video conferencing, video-phone, video on demand, and to some extent ípersonal Communication System èpcsè. A general structure for such video transmission system may consist of following components: input and output of video, a encoder and decoder to compress and decompress video data, buæers to store video data before transmission or after reception, and a transmission channel. Fig. 1.1 shows the generic structure of such video communication system. Video input device Video output device Encoder Decoder Encoder buffer Decoder buffer Transmitter (channel coder, error control) Receiver Transmission channel Figure 1.1: Structure of a generic video communication system. Unlike other type of digital data transmission, video transmission is subject to certain time and delay constraints because each compressed video frame has to be transfered to the decoder before the time it is scheduled to be decoded. Due to such constraints, a much higher level Quality of Service èqosè in terms of the guaranteed 3

16 transmission bandwidth, bounded delay and delay jitter, is required to support the video transmission. A distinguishing property of video transmission is the high data volume and Variable Bit Rate èvbrè nature of the video data stream. The bursty nature of the video data stream may cause severe degradation on the entire network service if sudden increases in the video data traæc occur. Hence, certain bit-rate constraints on the video data are required. Rate control is also necessary in the Constant Bit Rate ècbrè transmission, e.g., TV broadcast or cable TV, where the variable bit rate data stream of the video information has to be regulated in order to comply with the channel bandwidth and buæer constraints. The VBR nature of video data stream can be attributed to the fact that data compression techniques such as quantization and variable length coding are commonly used in most video codecs. The number of bits that is required for encoding each video frame èwhich will be referred as ëencoding rate" in our later discussionè is not constant throughout a video sequence. However, most video codecs have builtin adjustable compression parameters to increase or decrease the encoding rates in order to accommodate the diæerent transmission bandwidths of the various types of channels. Given this æexibility in selecting the operating encoding rates, the rate control can also be used to regulate the bit stream of the video data from the source encoder in a way such that the external constraints imposed by the transmission channel and application can be complied with. In this chapter, we will brieæy introduce some commonly used video compression standards. We focus our study on video transmission over VBR channels, therefore in Section 1.4 we will investigate the properties of some VBR channels that will be used in our later study of video transmission. 4

17 1.2 Video Compression The standardization of digital video compression is still evolving. In addition to MPEG-1, MPEG-2, H.261 and H.263 standards discussed above, other video compression standards such as MPEG-4 or the extension of H.263 ë7ë will be ænalized in the near future. Without precluding the use of other compression schemes, the compression standards mentioned above èi.e., MPEG-1, MPEG-2, H.261 and H.263è are transform-based coding systems which use 2-D Discrete Cosine Transform èdctè ë8ë to transform spatial image data into a frequency domain representation, in which most energy of the signal is compacted in the low frequency coeæcients. Distortion + Video input - DCT Quantizer Quantization scale Dequantizer Entropy encode Encoding rate Inverse DCT + + Motion compensation Frame buffer Figure 1.2: Block diagram of a general DCT-based encoder. Those DCT-based coding systems can be divided into the following steps: DCT transform of the image, quantization of the resulting coeæcients, and entropy coding of the quantized coeæcients ë9, 10ë. Fig. 1.2 depicts a general DCT-based encoder. DCT coeæcients are unequally quantized by a quantization matrix which is specially designed according to the spectral response of the human visual system. 5

18 A Further degree of compression on video signal can be achieved by reducing the temporal redundancy between consecutive video frames, which is often achieved by Motion Compensation èmcè ë11, 12, 13ë. In video compression with MC, video frames are predictively encoded by using motion vectors to indicate the translation of each sub-divided region in a predicted frame from other frameèsè ë14ë. The process to search for such motion vectors is called Motion Estimation èmeè. In the MPEG standards, for example, multiple prediction methods èi.e., predictive coding from previous frames and bidirectional coding from previous and future framesè are used to eæectively reduce the prediction error in ME. After MC, only the resulting prediction errors and motion vectors have to be encoded. Video frames are classiæed into three picture types depending on which prediction methods are used: Intra frame èi-frameè which is independently encoded without MC; Predicted frame èp-frameè which is unidirectionally predicted from previous frame; and Interpolated frame èb-frameè which is bidirectionally predicted from previous and future frames. Fig. 1.3 depicts an example that video frames are are encoded in diæerent in picture types in a predictive video coding scheme Variable quantization The video codec discussed above is a ëlossy" compression approach because the quantization of the DCT coeæcients induces noise in the reconstructed video. While the introduced distortion in the reconstructed video may be imperceptible or tolerable by the users, video encoding by lossy compression schemes can achieve much higher compression ratio than that of the lossless compression schemes. One propertyofthe lossy compression approaches is that the video encoding rates usually are scalable in those approaches. In the DCT-based compression scheme discussed above, for 6

19 Forward prediction Video sequence display order I B B P B B P B B I Intra-frame (I-frame) Backward prediction Forward predicted frame (P-frame) Bidirectionaly predicted frame (B-frame) Figure 1.3: Video frames that are encoded in diæerent picture types in a predictive video coding scheme. example, the resulting encoding rate and the associated distortion of a video frame are dependent on how coarsely the DCT coeæcients are quantized. The coarseness of the quantization can be universally scaled by adjusting the quantization step size èe.g., the MQUANT parameter in MPEGè. Therefore, varying the encoding rate can be achieved by adjusting the quantization scale at the encoder. Nevertheless, other encoding parameters, such as the selection of the picturetype and macroblock-type, also aæect the resulting encoding rate and distortion, and rate control also can be achieved by mode selection ë15ë. In general, the distortion increases as the encoding rate decreases, and vice versa. Our goal will be selecting the encoding rate for each frame so as to minimize the overall distortion of the whole video sequence, given that the encoding rates are restricted by external system parameters, such as delay and channel constraints. 7

20 1.2.2 Other encoder functionalities for video transmission In designing video codecs for video transmission applications, other factors related to the communication channel, such as the transmission delay and errors, need to be taken into account. MPEG-2 has potential use in video transmission applications, hence several types of scalability such as SNR, temporal and spatial temporal scalability are deæned in the MPEG-2 standard ë4, 16, 17ë. In the recent development of the ITU Recommendation H.263 version 2 ë7ë, which is the extension of H.263 and usually referred as ëh.263+", several coding modes have been deæned to provide more functionalities for real-time transmission: such as the ëreference Picture Selection mode" which uses a feedback acknowledgement from the receiver to inform the encoder if a picture has been degraded by transmission errors so other unaæected pictures can be used as the reference frame to predict following frames; the ëtemporal, SNR and Spatial Scalability mode" in which the quality of the transmitted video can be adjusted according to the limitations of channel throughput and other conditions by selectively transmitting the enhancement data on top of the baselinequality video data; and the ëindependently Segmented Decoding mode" to conæne the error propagation. Other modes such as ëslice Structured mode" deænes the slice structure containing variable numbers of macroblocks. The use of slice structure allows more æexibility in sub-dividing the picture, which can be more suitable for the underlying packet transport, more eæcient in error resilience and result in lower encoding delay. 8

21 1.3 Rate Control for Video Transmission In the video communication system shown in Fig. 1.1, buæers are used in both the encoder and decoder to temporarily store the coded video data before the transmission or after the reception. In a CBR transmission environment, buæer between the video codecs and transmission channel can smooth out the variable source rate generate a constant output rate to the channel ë18ë, whereas in a VBR transmission, buæer can function as ëtraæc shapper", which smoothes out the bursty video bitstream and alleviates the potential traæc burden of the transmission channel or network. In most video applications, the physical buæer sizes are limited, and a rate control mechanism is required to regulate the source encoding rate in order to prevent buæer overæow and underæow. In MPEG, a hypothetical Video Buæering Veriæer èvbvè decoding model èmpeg-2 Annex C ë4ëè has been deæned to constrain the source encoding rate. The rate control at the source coder constrains the video encoding rate in order to prevent overæowing or underæowing the hypothetical decoder buæer, which otherwise could be problematic for the decoder to receive and decode video data. It is usually desirable to keep a more consistent video quality for the encoded video sequence. Therefore in addition to prevent buæer overæow or underæow, another goal that can be achieve by rate control is to properly allocate bit among video frame or within a frame for consistent or better reconstructed video quality. The rate control algorithm Test Model 5 ètm5è ë19ë in MPEG uses the feedback from the buæer occupancy to modify the source encoding rate, and allocates bits to diæerent frame types, namely I-, P-, and B-frame, proportionally such that the reconstructed video has more consistent video quality over frames with diæerent frame types. 9

22 As discussed in previous section, in most video application a rate control mechanism is often required to constrain and regulate the encoding rate of the video, such that the bit rate of the compress video data stream can comply with external constraints that are imposed by the application and hardware. The importance of rate control is more evident in applications involving transferring video data in real time to the decoder through a communication channel, where video transmission is subject to the channel bandwidth and the delay constraints. While the major focus of rate control is on constraining the bit rate and avoiding the buæer overæow and underæow, the delay constraining factor is often overlooked by most rate control schemes. The rate constraints that are related to the delay constraints were formally formulated in ë20ë. An optimal rate control approach, in the sense of maximizing the video quality, for video transmission over CBR transmission environment with such delay constraint has been proposed in ë21ë. Extending this previous research, in this dissertation we study the rate control problems for video transmission over various VBR channels ë22, 23, 24, 25ë. Speciæcally we will focus our discussion on the video transmission over two types of VBR channels, namely broadband public access network such as ATM networks, and burst-error channel such as wireless links. First we brieæy introduce the characteristics of the VBR channels which are considered in our study. 1.4 Transmission Channels Due to the high acceptance and wide popularity of visual communication, many communication networks and channels have been investigated and studied as the platforms of video transmission. Traditionally CBR channels, such as circuit-switched 10

23 channels, dedicated connections with æxed transmission capacity, or a reserved bandwidth built on top of a packet-switching network ë26ë, have been used for video transmission. The reason for using CBR channels for video transmission is that the guaranteed transmission bandwidth and bounded transmission delay, which are essential for real-time video transmission, can be more easily achieved. However, as the emerging broadband public access networks become more popular and accessible, these broadband networks with VBR transmission capability are regarded as a major platform for various video transmission application in the future. VBR transmission capabilities have the potential of bringing about substantial beneæts for video transmission: constant or consistent video quality can be achieved through VBR video coding and thus high quality real-time applications with bounded transmission delay may become feasible with VBR transmission ë22, 20, 27, 28ë. VBR video transmission in a network environment can result in potential gains in network utilization, the so-called Statistical Multiplexing Gain èsmgè, when multiple VBR video sources are multiplexed ë29, 30ë. Shorter transmission delay can be achieved by using VBR transmission instead of CBR transmission, because in a CBR system extra data buæering, which may cause extra delay, is required when the æxed channel capacity cannot accommodate the variations of video source bit rate ë31ë. In our research of video rate control, we focus on the following two types of transmission channels: èiè broadband networks with usage parameter control, for example transmission of Video over ATM, and èiiè unreliable channels with burst transmission errors, for example Internet or wireless channels. The characteristics of these two types of channels are described brieæy in the following section. A more complete study of the rate control problem over these channels will be presented in Chapter 3 and Chapter 5. 11

24 1.4.1 Video transmission over broadband networks Future public access networks are expected to provide transmission services for various forms of information, such as data, audio, and video. Asynchronous Transfer Mode èatmè networks are an example of a network architecture designed for broadband applications in public networks. ATM networks have been increasingly used for real-time multimedia transmission ë32, 33, 34ë as their design is inherently applicable to provide fast and reliable transmission with predictable bandwidth provision and delay bounds. The ATM Adaptation Layer èaalè speciæcation ë35, 36ë deænes a set of connectionoriented VBR transmission services to be provided by the ATM networks, which are suitable for real-time video applications. In addition to the VBR transmission capability, real-time video transmission also requires guarantees on bounded transmission delay and delay jitter, the so-called Quality of Service èqosè. The QoS guarantees ensure that real time display of video at the decoding end is possible. The QoS of the network can be maintained if the traæc æow in each connection is monitored and regulated by a ëpolicing function" or ëusage parameter control". Policing functions enforce traæc æow control and ensure that the source traæc æow from each user complies with the negotiated usage parameters. In order to eæciently monitor the arriving traæc and respond to any violation of usage parameters, most of the policing functions will be simple and very easy to implement ë37, 38ë. Examples of simple policing mechanisms include leaky bucket, double leaky bucket, jumping window, sliding window, etc., and will be brieæy discussed in Chapter 3. We will focus on the study of video transmission over ATM networks with policing functions that are monitored at the user-network interface and are known to the video encoder. Video encoding rate thus can be controlled by the encoder with 12

25 a desirable goal being to avoid the violation of any applicable policing function constraints Video transmission over burst-error channels Research on video transmission over wireless channels has also become popular, because wireless connection can provide convenient tetherless data access and mobility to users. Video communication over wireless is challenging as the hardware and channel resources are very constrained. Given the bandwidth limitation of the wireless links, an eæcient video compression is often required for wireless video applications. In addition, compared to other wired transmission channels, wireless links suæer from limited bandwidth and are more likely to see their performance degrade. The compressed video data stream is vulnerable to transmission errors due to the predictive coding and entropy coding schemes that are often used in video codecs. Therefore, reliable transmission of highly compressed video signals over error-prone wireless channels is the more signiæcant issue of the wireless video transmission. A special characteristic of transmission errors in the wireless channels is that errors tend to occur in bursts. Therefore error control techniques such as Forward Error Correction èfecè would require large overhead in terms of redundancy to eæectively correct all the clustered errors. However, such overhead is a waste of transmission bandwidth during the period when the channel is in good condition. Recent research has considered ways of improving the transmission reliability by making use of the feedback channel for ëclosed-loop" error control, including various forms of retransmission ë39, 40, 41ë. Using the feedback channel for error control has been accepted by ITU-T as part of the mobile extension of H.263 Recommendation ë7ë. Additional delay may be introduced in the feedback and retransmission 13

26 process, which may be problematic in real-time applications with critical delay constraints. Rate control in this delay-constrained video transmission over error-prone channels thus focuses on regulating the bit rate of video data stream to cope with the dynamically change channel conditions and meet the delay constraints. We will present a rate control scheme for robust video transmission over burst-error channel in Chapter Overview and Contribution In Chapter 2 we formulate the problem of rate control in a video communications system by taking into account all the available resources and constraints. We introduce the concept of eæective buæer size as the single constraint on encoding rate combines the system delay constraint and the channel bandwidth constraint. This eæective buæer size can be seen as a convenient representation of the constraints the encoding rate has to meet in order to comply with the delay constraint and channel constraint, and will be the basis of our rate-control algorithms for video transmission. In Chapter 3 we focus on video transmission over ATM networks. In this environment both source bit stream and transmission data æow are VBR, and the source encoding rate and channel transmission rate can be controlled by the rate control mechanism. By combining the delay and channel constraints as in Chapter 2, an algorithm based on dynamic programming is proposed to jointly select the video encoding rate and transmission rate which can comply with these imposed constraints and achieve minimum video distortion. This joint selection algorithm can also be used as a benchmark tool to provide a quantitative comparison of the video quality enhancement gains when using a VBR channel instead of a CBR one. 14

27 In Chapter 4 we consider a real-time rate control scheme for the applications in which video is captured and encoded just before transmission, a scenario that arises in applications such as video conferencing or videophone. Encoding rate constraints in this real-time system are formulated with the goal of delivering video information in time for decoding. Algorithms based on dynamic programming and Lagrangian optimization for optimal bit allocation are proposed. This real-time rate control approach is also the basic structure for the robust video transmission over unreliable channels introduced in Chapter 5. In Chapter 5 we look at the problem of video communication over a bursterror channels. A new scheme which combines the rate control and the error control elements in a communication system is proposed to achieve robust video transmission over an unreliable channel. We focus on the ëtwo-way" wireless channel where a feedback channel exists, and an error control scheme like Automatic Repeat request èarqè can be used for data re-transmission. It can be shown that the overall robustness of video transmission can be improved through the source rate control assisted by an appropriate channel model and real-time feedback of the channel condition. Conclusions of our research are summarized in Chapter 6. 15

28 Chapter 2 Video Transmission 2.1 Overview of Digital Video Applications Nowadays, digital video is used in a wide range of multimedia and communication applications to disseminate information. Due to the diversity in the nature of those applications, the underlying system and hardware requirements may bequite diæerent, even though most of the video transmission application share a similar structure as shown in Fig For example, high video quality is of prime importance in the system design for digital video storage applications èvcd, DVDè or HDTV applications, whereas low delay latency èto make interactive visual communication possibleè is the key concern in video-conferencing applications. Therefore, in addition to the core video codec component, other system-level components ë34ë, such as rate control, channel error control and networking interface, have to be adjusted to the speciæc needs of diæerent applications. In order to get more insight of various constraining factors in a video transmission system, which are required in our research to develop the corresponding rate control approaches, we ærst classify video applications into the following three types èsee Fig. 2.1è according to where 16

29 the source video data is initially located and how video data is transferred to the decoder: I. Pre-compressed video is available on a local storage device: In this type of applications, video data is retrieved from a local storage device. The bandwidth of the internal data bus is considered to be suæciently large to handle the most demanding video data transfer rates. Examples of such applications include Video CD, Digital Versatile Disc èdvdè movies, or any video applications in which the data of the whole video sequence has been transferred to the local storage device before displaying the video sequence. II. Real-time transmission of pre-compressed video data: Video is pre-compressed and stored in a remote site. When the video displaying session begins, video data is transferred through a communication channel to the decoder followed by the decoding and display processes immediately or with a small decoder buæer delay. One important requirement for this realtime transmission is that the video data of each frame has to be received at the decoder before the time for decoding. Therefore there is a delay constraint in transferring the video data, and the applicable video encoding rate is also constrained by the communication channel bandwidth. In some applications a longer delay may be tolerable. In these applications the overall delay latency is only noticeable at the beginning of the video transmission session, i.e., the time it takes to display the ærst video frame. Examples of such applications are video on demand or broadcast of pre-recorded program. III. Real-time encoding and transmission In this type of application, video data is captured, encoded, and delivered to 17

30 the encoder in real-time just before transmitted to the destination. The overall delay latency thus begins from the time that a video frame is captured and encoded till the time that video frame is decoded and displayed. In the particular case of interactive applications, such as video conferencing and videophone, a short end-to-end delay is required. This is the case where delay constraint is much more restricted than that in the two previous cases. However in some examples such live TV or video broadcast, the delay constraints is not so restricted although video data is also encoded and transmitted in real-time. In these cases the delay requirement is more like ofthe type II application mentioned above, and the delay latency is perceived as the initial setup before the video session. In type I applications, video data is retrieved from the local storage device. There is no bottleneck in the data transfer between the video source and the decoder. The only possible constraining factor is the capacity of the storage device. Therefore the main purpose of the rate control in this type of applications is to meet the total bit budget, and to distribute bits evenly for encoding each video frame. In type II and III applications video data is streamed into the decoder through a communication channel in real time and is immediately decoded and displayed. We will refer to this type of data transfer as ëreal-time video transmission" in the later discussion. In these cases a large local storage device, as that in type I applications, is no longer needed. Instead only a small decoder buæer is required to temporarily store the arriving video data. However, other problems arise such as whether the channel bandwidth is large enough to support video transmission, or whether video data can be delivered in time for decoding and displaying. Since a communication channel is used to transmit the video data, the reliability and transmission delay of the channel 18

31 Local storage device Buffer Decoder Display Pre-compressed video data I. Pre-compressed video in local storage device Remote storage device Transmission channel Buffer Buffer Decoder Display Pre-compressed video data II. Real-time transmission of pre-compressed video data Video capture device Encoder Transmission channel Buffer Buffer Decoder Display Video data captured in real-time III. Real-time encoding and transmission Figure 2.1: Diæerent types of video applications also aæects the eæectiveness of real-time video transmission. Therefore, rate control in these types of applications should aim to regulate the bit-rate of video data stream to comply with those additional constraints that are imposed by the communication channel and the real-time requirement ofthe applications. 19

32 2.2 Delay Latency and Delay Constraint in Video Transmission As discussed above, delay constraint for the transport of video data is one of the major reasons for requiring a rate control mechanism at the video encoder. To formulate the delay constraint, ærst we deæne the end-to-end delay, denoted as æt, as the time interval elapsed from the time a video frame is captured and encoded at the transmitter until the time when the video frame is decoded and displayed at the receiver. In a real-time video communication system where both encoder and decoder are attached to synchronous devices ècamera and display, respectivelyè, the end-toend delay of a video frame traversing the system should be constant. Video data that arrives at the decoder too late to be decoded by its scheduled display time is useless and is considered lost. Clearly, frame skipping at the decoder results in quality loss, especially when motion compensated video coding is used, while skipping frames a the encoder can be done without as heavy a quality penalty. However in this latter case there will also be end-to-end delay constraints for those frames that are transmitted. Diæerent video applications have diæerent delay requirements. In interactive video communications èe.g., video conferencingè low delay is required, while in one-way video transmission èe.g., broadcast or video on demandè the endto-end delay is only noticeable to the user as an initial latency, i.e., the time interval between the start of the video transmission session and the time the ærst video frame is displayed. Frame loss may result if some of the information corresponding to a video frame arrives at the decoder after the scheduled decoding time. Since the information received at the decoder is stored in the decoder buæer before actually being decoded, we will call this situation decoder buæer underæow. 20

33 For a generic video communication system, the end-to-end delay æt may consist of the following major components èrefer to Fig. 2.2è as: æt = æt e èencoder delayè + æt eb èencoder buæer delayè + æt c èchannel transmission delayè + æt db èdecoder buæer delayè: + æt d èdecoder delayè: è2.1è Transmitting unit Receving unit Video input Video encoder Encoder buffer Channel Decoder buffer Video decoder Video output Delay: Te T eb Tc T db T d Figure 2.2: Delay components of a video communication system. The end-to-end delay ætin è2.1è has to be constant for the transmission of every video frame in order to keep synchronized timing between encoder and decoder. In most video codecs, a bounded maximum encoder and decoder delay is more likely to be expected, because the design of the encoder and decoder has to meet the minimum performance requirement that every frame can be encoded or decoded within each frame interval. We thus assume a constant encoder encoder delay æt e and the decoder delay æt d in our formulation. The channel delay æt c, however, could be variable in some transmission environments. In the point-to-point communication channel with direct connection, the transmission delay æt c can be treated as æxed because of the æxed delay latency 21

34 in the transmission link. However in public access networks, it may be diæcult for the networks to provide transmission services with æxed transmission delay due to the possible network congestion caused by unpredictable increases in traæc load. Therefore, we over-dimension the variable transmission delay æt c by its maximum expected value ætc max, given that the channel can provide data transmission service with bounded transmission delay. Excluding the delay elements æt e, æt d, and æt max c from the constant end-toend delay æt in è2.1è, we can focus on the buæer constraint due to the total buæer delay èat encoder and decoderè as: æt eb +æt db =æt,æt max c, æt e, æt d è2.2è Deæning the time interval for one video frame as T f, then the number of video frames that are stored in either encoder or decoder buæer is æn = æt eb +æt db T f = max æt, ætc, æt e, æt d : è2.3è T f What è2.3è states is that given a end-to-end delay constraint æt, there will be æn video frames stored in both the encoder and decoder buæer. æn can also be interpreted as the buæer delay in terms of the number of frames. From è2.2è if æt e,æt d, and æt max c are constant, then æn will also be constant. However we over-dimension the channel delay æt c by its maximum expected value æt max c in above formulations. If the actual channel delayæt c is smaller than æt max c, video data will arrive at the decoder buæer earlier. In this scenario, more data èor video framesè will be stored in the decoder buæer than that in our formulations, but the compliance with the end-to-end delay constraints can be ensured by assuming 22

35 the worst cast transmission delay, if the decoder buæer size is also over-dimensioned to accommodate those data that arrive earlier. Note that in what follows we consider that the encoder and decoder clocks are shifted by an amount equal to æt max c, the transmission delay. Thus, if the i-th frame interval starts at time t i at the encoder, it will start at time t i +æt c at the decoder. Refer to Fig 2.3. Encoder time index Decoder time index Frame 1 Frame 2 Time 1 Time 2 T max c Τ Video Sequence at Encoder... Delay Ν frames Delay Ν frames Frame i Frame 1 Frame Time i... Τ Video Sequence at Decoder Frame i... Time 1 Time Time 1+ N Time 2+ N Time i+ N Figure 2.3: Timing diagram of encoder and decoder, note that the clocks at encoder and decoder are shifted by æt c the transmission delay. 2.3 Encoding Rate Constraints Related to Delay Constraint We will now discuss how the video encoding rates are constrained by the end-toend delay, channel transmission rate, and the sizes of the buæers at the encoder and decoder. Denote B e èiè and B d èiè, respectively, the encoder buæer and decoder buæer occupancies at time i. Let Cèiè and Rèiè be, respectively, the channel rate at 23

36 time i èi.e., during the i-th frame intervalè and the source rate used for i-th frame. The formulation does not change if one considers a time unit smaller than the frame interval. For example, as will be seen in our experiments, we can use the Group of Blocks ègobè as the basic unit so that Cèiè and Rèiè become, respectively, the channel and source rates for the i-th GOB. All of these variables have units of bits. The encoder and decoder buæer occupancies can be obtained as B e èiè = B d èiè = ix 8 é j=1 é: Rèjè, ix j=1 Cèjè è2.4è P i j=1 Cèjè, P i,æn j=1 Rèjè; when i ç æn P i j=1 Cèjè; when iéæn. è2.5è Note that, because of the end-to-end delay, the decoder waits æn frame intervals before starting to decode the video frames available in its buæer. Here we are assuming suæciently large physical buæers at encoder and decoder and thus we focus on the eæect of buæer underæow. For most cases of interest, given that our goal is to minimize distortion, the set of quantizers will be such that encoder buæer underæow will not occur and we thus do not take this into account. However decoder buæer underæow, i.e., the situation where not all the bits corresponding to a given frame are present at the decoder in time to be decoded, may very well occur. This problem is important since it results in frame losses. Combining the encoder buæer occupancy è2.4è at time i and decoder buæer occupancy è2.5è at time i +æn,wehave that: B d èi +ænè = = i+æn X j=1 X i+æn j=i+1 Cèjè, ix j=1 Cèjè, B e èiè; Rèjè = i+æn X j=i+1 Cèjè, è ix j=1 Rèjè, ix j=1 Cèjèè è2.6è 24