Burst-Packet-Loss Concealment for MPEG-2 Video

Transcription

1 Burst-Packet-Loss Concealment for MPEG-2 Video Shih-Hsuan Yang and Jia-Ming Lin Institute of Computer, Communication, and Control National Taipei University of Technology 1, Sec. 3, Chung-Hsiao E. Rd., Taipei, Taiwan Abstract - In this paper, we propose an object-based error-concealment technique for MPEG-2 video transmitted in a burst-packet-loss environment. A burst packet loss typically impairs consecutive slices for the reconstructed picture. The proposed object-based error concealment proceeds in two stages, by intra-picture and inter-picture estimation respectively. In the first stage, regions of the missing area are estimated by using the motion vectors adjacent to the missing area. In the second stage, the singular objects with respect to uniformly moving background within the missing area are identified and compensated from the prediction pictures. As compared with conventional concealment approaches, the proposed method achieves better PSNR performance and relieves the visual discontinuity over the compensated region. I. INTRODUCTION The MPEG-2 coded video has proliferated owing to the wide adoption of digital video disks (DVD) and digital video broadcasting (DVB). The MPEG-2 video coding [1] is a hybrid DPCM/DCT lossy compression algorithm. Temporal redundancy of video with respect to reference pictures is removed by block-based motion compensation, in units of macroblocks. Spatial correlation of an intra-block or a residual block is extracted by performing DCT (discrete cosine transform), scalar quantization, and entropy coding. According to how temporal prediction is involved, three different picture types are defined in MPEG-2, namely I (intra), P (predictive), and B (bi-directional) pictures. The predictive nature and the use of variable-length coding (VLC) make MPEG-2 coded video very vulnerable to transmission errors. Error-resilient coding techniques have therefore been proposed to circumvent the severe quality degradation of using efficient source coding techniques over unreliable channels [2]-[6]. Error-resilient video coding mechanisms can be classified into three groups. First, the error-resilient encoding methods utilize joint source-channel coding to make the compressed stream more robust to avoid disastrous effects. Second, when a feedback channel is available, an interactive encoder may be adapted to network conditions to suppress error propagation. Finally, error concealment techniques estimate the erroneous or missing sample at the decoder by using the residual correlation between the lost data and the correctly received information. Among these three approaches, error concealment strategy is the most flexible since it does not require any modification to the syntax of the coded video and any special network environment. Error concealment for MPEG-2 coded video can be performed in a form of interpolation in either spatial domain or temporal domain [7]-[11], by making use of the available regions or pictures close to the damaged area. A large class of temporal-domain concealment techniques manages to restore the lost data by intelligently estimating the motion vector (MV) of the lost macroblock. The MPEG-2 video may be transmitted over an IP/UDP/RTP-based network for real-time applications, possibly in conjunction with a higher-level communication protocol such as H.323 [2]. Although the transmission is almost bit error free, high packet loss rate (up to 30%) may occur due to congestion. The packet loss is usually bursty in nature, which ruins several consecutive slices of a picture. Conventional error concealment techniques may be inefficient or difficult to implement in this scenario because they usually assume that local information around the lost macroblock is available [7]-[11]. In this paper, we develop a new two-stage object-based error concealment technique for MPEG-2 video over burst-packet-loss networks. Both stages employ temporal interpolation. In the first stage, the missing slices are estimated by available area in the corrupted picture and compensated using a region-based approach. In the second stage, irregularly moving objects with respect to the

2 background are identified from the predictive picture using a boundary detection technique. This two-stage strategy is based on the observations that there exists high temporal correlation between the decoded picture and its neighboring pictures, except for some singular objects. Experimental results show that our method provides superior performance to the conventional approaches in both objective and subjective quality measurements. The paper is organized as follows. The MPEG-2 coding process relevant to our work is summarized in Section II. The proposed method is addressed in Section III. Simulation results along with comparisons are provided in Section IV. Finally, we conclude our work in Section V. II. MPEG-2 CODING PROCESS We encode the CCIR-601 (NTSC) sequences at target bit rate 4 Mb/s, which generates high-quality picture for standard-definition digital video. We take a slice to be an entire row of macroblocks. Under the IP/UDP/RTP packet-based circumstances, video data are transmitted in packets with a variable packet size typically no larger than 1500 bytes. To accommodate the coded MPEG-2 video data, each transport data packet is loaded with 1/2, 1, and 3 slices for I-, P-, and B-pictures, respectively. As a consequence, a lost packet may result in a loss of 1/2, 1, or 3 slices. Decoding is resumed from the next available packet with the aid of slice headers that include the synchronization marker and the vertical position of a slice. We then follow the error concealment strategy below to compensate the missing area. III. PROPOSED METHOD Stage 1: Intra-picture compensation In most cases, there exist large regions (the background scenes) within pictures, which hold still or move uniformly in the course of a video sequence. Lost background scenes can be effectively compensated in an integral way, especially when they occupy a large portion of a picture. Instead of using the conventional macroblock-based approaches, we divide the missing area into regions such that each region has homogeneous motion. Each region is compensated as a whole from the nearest reference picture using the region s estimated motion vector obtained as described below. Motion vectors directly over and under the missing area are first median filtered (with window size 3(H) 1(V)) to remove the sporadic abrupt change. Let V x (i) and V y (i) denote the horizontal and vertical component of the i th filtered motion vector, we then calculate the gradient of motion vectors (GMV): GMV(i) = (I 2 x (i) + I 2 y (i)) 1/2, (1) where I x (i) = V x (i+1) V x (i), (2a) I y (i) = V y (i+1) V y (i). (2b) A large GMV (larger than 2.5 (average of the associated GMVs) in our simulation) indicates the boundary of a region. The missing area is thus vertically divided into rectangular regions. We take the median of the associated motion vectors bordering a region as the estimated motion vector, and perform the region-based compensation from the nearest reference picture. In case that the motion vectors are not available (e. g., for intra-coded macroblocks), we perform motion estimation at the decoder before error concealment to obtain the required motion vectors. Our result is demonstrated in Figure 1, where the coded data of the shaded area in Figure 1(b) are lost. The missing area is divided into three regions using our region-based segmentation: the trunk, and the background scenes to the left and to the right of the trunk. The first-stage concealed result is illustrated in Figure 1(c) and it can be seen that the background is successfully interpolated. Compared with the results obtained by conventional macroblock-based approaches shown in Figure 1 (e) and (f), this region-based approach effectively reduces blurs and blocking effects. The concealed trunk, however, exhibits severe misalignment and requires further processing. Stage 2: Inter-picture compensation The intra-picture compensation fails to give satisfactory results when there exist irregular-motion objects with respect to the background. The second-stage inter-picture compensation extracts these objects from the predictive picture by boundary detection. The predictive picture is the nearest P picture to the corrupted picture except that for the video sequence in the display order of IB BP the first B picture will be used as the predictive picture of the P picture. We use the boundary detection algorithm given in [12] for object identification, and adapt it from the pixel-based approach to our MV-based scenario. The search range for objects in the predictive picture is centered at but larger than the corresponding missing area in the corrupted picture. Two feature functions of motion vectors

3 are employed for boundary identification: the Laplacian zero-crossing function f Z (.) and the gradient magnitude function f G (.). The Laplacian zero-crossing function f Z (p,q) is a binary-valued function that equals 1 when the second-order derivative crosses zero at position (p,q) and equals 0 otherwise. The gradient magnitude function is defined as f G (p,q) = 1 G(p,q)/max(G), (3) where G(p,q) is the two-dimensional GMV at position (p, q). These derivatives are calculated similar to what we did in (1) and (2) but are extended to two dimensions. The local cost of a macroblock located at position (p, q), denoted by l(p, q), is evaluated as l(p,q) = w Z f Z (p,q) + w G f G (p,q). (4) Empirically we take the relative weights w Z and w G to be both 0.5 in our simulation. Appropriately selecting the initial point of a potential boundary (which has a small local cost), we then use dynamic programming to locate the path with the minimal average cost as the object boundary. A path is going downwards to reach the lower border of the search region. The average cost C(.) of a path Ω equals the weighted sum of the local costs it traverses. In other words, C( Ω) = [ l( i, j ) / len( i, j) ] ( i, j) Ω (5) where len(i, j) is the length from the previous point on the path to (i, j) (i.e., a diagonal segment is scaled by 2 ). A closed boundary or two boundaries with relatively small GMVs in between form an object. The identified object is compensated from reference pictures. The required motion vector for the object is properly scaled by the temporal relations of the corrupted picture relative to the predictive picture and the reference picture. The progress of the second-stage concealment is illustrated in Figure 1(d). It is clear that the misalignment of the trunk is much relieved after the proposed object-based inter-picture concealment. IV. EXPERIMENTAL RESULTS The presented concealment scheme is evaluated using three MPEG-2 test sequences (150 frames each): Flower Garden, Mobile & Calendar, and Stefan, all in the CCIR-601 4:2:0 NTSC format. We approximate the packet loss process by a two-state Markov model known as the Gilbert channel [13]. The average packet loss rate (PLR) of the channel is set to be 5% and the average burst loss length is set to be 2.55, 3.4, and 5.1 packets, respectively. Our scheme is compared with two known methods in the literature [8], and the results are given in Table I and Figure 1. The block replacement method replaces the lost macroblock with the corresponding macroblock in the nearest reference picture. The median MV method takes the median of the available motion vectors around the lost macroblock for error concealment. Only the luminance component (Y) is taken for the PSNR (peak signal-to-noise ratio) calculations. In the burst-loss environment the new method is superior to the median MV method (the better of the two conventional approaches) by 0.2 to 1.5 db, and it provides comparable performance in the random-loss environment. The perceptual improvement is significant by viewing the sequence or looking at the pictures in Figure 1. It is also revealed from the simulation conducted on a Pentium-III 650 PC that the proposed error concealment requires extra 30% of the decoding time, where a larger portion (about 70%) is devoted to inter-picture compensation. A practical MPEG-2 decoder can therefore benefit from this method with only moderate increase in complexity. V. CONCLUSION In this paper, we propose a new two-stage error concealment technique for MPEG-2 coded video over burst-packet-loss networks. In the first stage the background scenes with homogeneous movement are compensated using adjacent motion vectors in the corrupted picture. The second-stage compensation locates the irregular-motion objects in the missing area from the predictive picture using a boundary detection approach. Our method provides significant improvement over conventional error concealment schemes not aimed for burst packet losses. VI. ACKNOWLEDGEMENT This work is supported by the National Science Council, R. O. China, under the contract number NSC E REFERENCES [1] ISO/IEC , Information Technology Generic Coding of Moving Pictures and Associated Audio Information Systems, Part 2: Video, [2] Y. Wang, S. Wenger, J. Wen, and A. K. Katsaggelos, Error resilient video coding

4 techniques, IEEE Signal Processing Magazine, vol. 17, no. 4, pp , July [3] Y. Wang and Q.-F. Zhu, Error control and concealmet for video communication: a review, Proc. IEEE, vol. 86, no. 5, pp , May [4] Special Issue on Error-Resilient Image and Video Transmission, IEEE J. Select. Areas Commun., vol. 18, June [5] Special Issue on Error Resilience, Signal Processing: Image Commun., vol. 14, May [6] P. Frossard and O. Verscheure, AMISP: a complete content-based MPEG-2 error-resilient scheme, IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 9, pp , Sept [7] S. Tsekeridou and I. Pitas, MPEG-2 error concealment based on block-matching principles, IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, pp , June [8] J. Zhang, J. F. Arnold, and M. R. Frater, A cell-loss concealment technique for MPEG-2 coded video, IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, pp , June [9] S. Shirani, F. Kossentini, and R. Ward, A concealment method for video communications in an error-prone environment, IEEE J. Select. Areas Commun., vol. 18, no. 6, pp , June [10] P. Salama, N. B. Shroff, and E. J. Delp, Error concealment in MPEG video streams over ATM networks, IEEE J. Select. Areas Commun., vol. 18, no. 6, pp , June [11] Jill M. Boyce, Packet loss resilient transmission of MPEG video over the Internet, Signal Processing: Image Commun., vol. 15, pp. 7-24, Jan [12] E. N. Mortensen and W. A. Barrett, Intelligent scissors for image composition, Proc. of the 22 nd annual ACM conference on Computer Graphics, 1995, pp [13] E. N. Gilbert, Capacity of a burst-noise channel, Bell Syst. Tech. J., pp , Sept

5 中文摘要 在本論文中, 我們提出 MPEG-2 編碼視訊在叢集封包遺失環境下的錯誤隱藏技術 叢集封包遺失通常會造成 MPEG-2 畫面較大範圍的損毀 我們所提議的錯誤隱藏技術以物件為基礎, 並分畫面內與畫面間兩階段進行 在第一階段中, 我們先利用損毀畫面正確解碼區塊的移動向量分區補償損毀區域的背景 第二階段則藉由預測畫面找出相對於背景的不規則移動物件, 再利用畫面間的相關性予以重建 與傳統的錯誤隱藏技術相比, 我們所提出的方法不但有更高的 PSNR 值, 也能減少補償區域畫面不連續的瑕疵 TABLE I AVERAGE PSNR (IN db) FOR THE ERROR CONCEALMENT METHODS UNDER STUDY (PACKET LOSS RATE = 5%) Error-free 28.1 Average burst loss length random loss Flower Garden No concealment Block replacement Median MV Proposed method after Stage Proposed method after Stage Error-free 27.7 Average burst loss length random loss Mobile & Calendar No concealment Block replacement Median MV Proposed method after Stage Proposed method after Stage Error-free 29.5 Average burst loss length random loss No concealment Stefan Block replacement Median MV Proposed method after Stage Proposed method after Stage

6 (a) (b) (c) (d) (e) (f) Figure 1. Part of Frame 7 (P-frame) of the Flower Garden sequence: (a) error-free; (b) erroneous, PLR = 5% and average burst length = 3.4; Concealed by (c) our method after Stage 1; (d) our method after Stage 2; (e) block replacement; and (f) median MV.