ROBUST REGION-OF-INTEREST SCALABLE CODING WITH LEAKY PREDICTION IN H.264/AVC Qian Chen, Li Song, Xiaokang Yang, Wenjun Zhang Institute of Image Communication & Information Processing Shanghai Jiao Tong University Shanghai 200240, China {qianchen,song li,xkyang,zhangwenjun}@sjtu.edu.cn ABSTRACT We propose a region-of-interest () coding framework based on leaky prediction (LP) for robustly transporting H.264 video over error-prone network. The LP-based coding can remove error drift caused by the destruction in the decoded background from the proposed system, with guaranteed quality throughout. Experimental results show that this scheme enables the decoder to reconstruct the with better quality and the global video frame with improved quality even if the background bitstream cannot be correctly received or completely lost. Index Terms leaky prediction, region-of-interest, error resilience, H.264/AVC 1. INTRODUCTION Video coding applications over the Internet and wireless networks have gained significant interest. In such an environment, it is very important to make the coder adaptive to varying network throughputs to obtain good visual quality with the available rate resources [1]. An enable approach to cope efficiently with this challenge is the layered or scalable coding scheme. Region-of-interest () scalability is of crucial interest in application scenarios where some visual regions are more important or interesting than the other parts of the video image. Although H.264/AVC is finalized without support of scalable coding except for temporal scalability with hierarchial B frames [2], scalable coding can be easily implemented by using flexible macroblock order (FMO) [3]. Fig.1 illustrates how scalable coding can be achieved in H.264/AVC. is first defined in a video frame to divide the frame into area and non- area, and then each of the area is mapped to a distinct slice. Here area is mapped to slice 0 as layer, while non- mapped to slice 1 as background layer. The two slices can be treated differently This work was supported in part by National Natural Science Foundation of China (60332030, 60502034, 60625103), Shanghai Rising-Star Program (05QMX1435), Hi-Tech Research and Development Program of China 863 (2006AA01Z124), NCET-06-0409, and the 111 Project MB number 0 1 2 3 4 5 6 7 8 9 10 11 12 1314 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 8182 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 Background slice NAL Background slice NAL Fig. 1. scalable video coding using FMO in H.264/AVC, and single is defined in transportation [4]. Since layer is prior to background layer, it can be strongly protected by unequal error protection or using a reliable transport mechanism. However, the receipt of the background layer data packets is not guaranteed because they may be discarded partially or completely due to bandwidth variation. For scalable stream transmission over error-prone network, there is no guarantee that the layer can be correctly decoded. The reason is that due to transmission error, the background layer at the decoder may mismatch for that at the encoder in motion-compensation (MC) loop, and may be employed as reference in decoding the current layer. As shown in Fig. 2, when the ordered stream carrying the update information of background layer for f(n 2) is truncated, the reconstruction of background layer in f(n 1) will be falsified (marked by shaded area). Since f(n) might refer to background layer of f(n 1) in both and background layer decoding, the errors spread in the decoded f(n) (marked by shaded area in both layers). Even if all the following bitstream can be correctly received, the errors will propagate spatially and temporally to the following frames f(n+1), f(n+2) until the next intra-coded frame is received. To refrain background errors propagation, error concealment techniques[5]-[6] can be employed to hide visible distortion. However, the error remaining after concealment propagate to successive frames and remain visible for a longer period of time, which makes the resulting artifacts particularly annoying[7]. Another possible method to avoid background
f n 1 f n f n 1 Fig. 2. Error propagation in coding (shaded area indicates error, arrow indicates referring to previous frame) error propagation is to constrain the motion search into area, preventing inter-frame dependency between and background layers. Despite the coding robustness of constrained motion estimation (ME), it is scarcely used due to the inefficiency. In this paper, we propose a novel approach for scalable coding framework by introducing leaky factor in MC loop, so that any error propagated from non- area to the is damped. The proposed scheme has two options for the prediction loop: the whole frame and the layer. It partially employs a reference block from a constrained ME in layer, and partially from unrestricted ME in the whole reference. A leaky factor is deployed to weigh the two predictions to trade off the two reference blocks. If leaky factor α = 1, the layer is completely excluded from MC loop, yielding to the best efficiency but the worst error resilience performance. When α = 0, it restricts the ME range into area only and thus results in poor prediction with strong robustness. The rest of the paper is organized as follows:section 2 describes the proposed scalable coding with leaky prediction. In Section 3 we analyze the principle of error mitigation in layer with the proposed scheme. Section 4 gives the simulation results with analysis,which demonstrate the robustness of the proposed scheme. And Section 5 concludes the paper. 2. SCALABLE CODING FRAMEWORK WITH LEAKY PREDICTION 2.1. Encoder Structure Leaky prediction is a well-known technique to increase error robustness by balancing coding efficiency and robustness [8]. Since the leakage introduced by spatial filtering in a motion compensated predictor is not strong enough for error resilience [9] in region, which is considered much more important than the rest of the frame, we apply leaky factor α (0 < α < 1) to scale the whole reference employed by layer to accelerate the error degradation. The encoder framework of the proposed scheme is shown in Fig.3, by modifying the hybrid block-based coding structure in H.264/AVC. The difference lies in ME and MC loop. For a macroblock (MB) in slice, there are two prediction reference sources: the whole reference frame and the layer. An unrestricted ME is done in the whole frame to obtain the optimal prediction block. At the same time it conducts constrained ME within the decoded layer to get the second prediction block. The two blocks are then scaled by α and 1 α respectively, so as to generate a weighted sum as the final prediction for the coding block. The prediction of layer MB in f(n) denoted f (p) (n) is formulated as: f (p) (n) = αmc 1(f (r) ) + (1 α)mc 2 (f (r) ) (1) f (r) is the prediction block from the whole frame reference, f (r) is the prediction from layer reference, MC( ) denotes block motion compensation. It is worth noting MC 1 and MC 2 deploy two distinct motion vectors, MV 1 and MV 2 respectively, and both motion vectors have to be sent to the decoder. Basically, we try to put higher priority to the decoded layer quality over background layer than the background. In this paper, no leaky prediction is used in coding background slice, i.e.only the usual coding framework with unrestricted ME in the whole reference frame is carried out for background slice coding. Accordingly, the error mitigation speed in background layer is slower than that in layer due to the lack of extra leakage. Of course, both and background slices can be coded with leaky prediction. In this sense, the encoder framework in Fig.3 is not restricted to slice use, hence we do not distinguish the coding scheme utilized by the two slices in the proposed scheme. 2.2. Decoder Structure The leaky prediction decoder structure, shown in Fig.4 employs both the decoded overall frame and layer in the MC loop. The decoder receives two different sets of side information. The first set of block mode with MV 1 gets prediction from the whole frame, and the second mode with MV 2 is to obtain the prediction from the layer. Then the same gain factors α and 1 α are introduced to scale the two predictions respectively to get a combined prediction block for current decoded MB. Both slice and background slice MB can be decoded by this decoder if the MB is coded with leaky prediction at the encoder. For those MB not coded with leaky prediction, the conventional MC loop without layer reference is utilized in decoding instead. 3. ERROR MITIGATION WITH LEAKY PREDICTION In scalable video coding in H.264, the mismatch of background layer causes error to spread spatially across layer, yielding annoying artifacts in. A factor α less than 1 can severely attenuate the erroneous prediction signal from background, to mitigate the effect of the error in the decoded
_ + + 1 + Fig. 3. Encoder structure in scalable coding with leaky prediction + + Fig. 4. Decoder structure in scalable coding with leaky prediction following frames, while still preserving efficiency. In this section, we analyze the error robustness of scalability with leaky prediction. And the efficiency problem will be discussed in Section4. For a clear formulation of error propagation with leaky prediction, we simplify the scalable coding by the following three assumptions: 1) Particular care is only given to quality of the decoded layer, while error propagation in the non- region can be tolerated. Thus only macroblocks in slice deploy leaky prediction. 2) The erroneous case is specialized so that the background layer of only one frame is influenced by transmission error. Suppose it is f(n 1), and the bitstream for frames that follow f(n 1) is spared from error. 3) Single reference is used, and each frame only predicts from its previous one frame. Suppose is the error in the background layer of f(n 1). For each MB in slice of f(n), the prediction is f (p) (n) = αmc 1(f (r) (n 1) + ) +(1 α)mc 2 (f (r) (n 1)) (2) The optimal prediction f (r) (n 1) obtained by the unrestricted ME can be either inside, partially inside or entirely outside the previous layer. Here we assume the worst case, in which the optimal prediction is entirely in the non- area, i.e. Therefore f (r) (n 1) = f (r) non (n 1) (3) f (p) (n) = αmc 1(f (r) non (n 1) + ) +(1 α)mc 2 (f (r) (n 1)) = αmc 1 (f (r) non (n 1)) + α +(1 α)mc 2 (f (r) (n 1)) (4) For simplicity, 4 can be rewritten as where i(n) = αo(n 1) + e(n) + (1 α)i(n 1) (5) i(n) = f (p) (n) o(n) = MC 1 (f (r) non (n)) As an iterative expression of i(n) and i(n 1), and e(n) is the layer error in f(n), here e(n) = α. Since the leakage introduced by 1/4-pel accuracy in the MC loop, the transmission error in non- area mitigates with time step quantized by δ(t) according to the analytical model in [7], where δ(0) = 1 and 0 < δ(t) < 1 when t > 0. Therefore, the background error that spreads to layer in f(n+1) can be expressed as
0 e(n + 1) = α δ(1) + (1 α)α (6) Similarly, for f(n + t) e(n + t) = α δ(t) + (1 α)α δ(t 1) + Clearly, +(1 α) (t 2) α δ(1) + (1 α) (t 1) α α + (1 α)α + +(1 α) (t 2) α + (1 α) (t 1) α (1 α)t = α α = (1 α) t lim (1 t α)t 0 (7) Hence the error introduced from background decays over time and finally converges to 0 in the layer of the video sequence, faster than that in the background layer which decreases with δ(t). Though in true application, continuous errors in a bistream is more likely instead of a single frame error, the extra leakage proposed in the scheme also attributes to better error mitigation performance, as is demonstrated in Section 4.1. 4. SIMULATION RESULTS AND ANALYSIS PSNR(dB) anchor proposed 0 1 2 3 4 5 6 7 8 Fig. 5. PSNR loss of the decoded overall pictures and the layers in foreman QCIF (α = 0.1, BK = 1)when only the background layer of the first P frame is lost PSNR(dB) 0 We present experimental results in this section to illustrate the robustness and efficiency of the proposed scalable coding with leaky prediction by incorporating the proposed algorithm into H.264/AVC software H.264/AVC software JM9.8[10]. Packet loss simulation is conducted to compare the error resilience of the proposed scheme with that without the leaky prediction. The parameters related to the proposed scheme is investigated to analyze their respective influence on the performance. proposed -16 0 2 4 6 8 10 12 14 16 18 20 Fig. 6. PSNR loss of decoded layer and overall video in foreman QCIF (α = 0.1, BK = 1)when continuous errors occur periodically in every 50 frames 4.1. Overall Performance of Leaky Prediction in scalable Coding Based on the theoretical analysis in Section3, the error from background impairment decays over time and can finally converge to 0 in the layer of the video sequence, if the three assumptions have been satisfied. In Fig.5, we use the test sequence foreman (QCIF,12.5Hz,100 frames, QP=28, IPPP mode, single ) in the simulation. In this paper, we simulate a simple scenario that the background slice packets of the first P frame is discarded completely during transmission. We replace the corrupted image content by the corresponding pixels from previous frame as a simple approach for error concealment, which yields good results for sequences with little motion[6]. It can be clearly observed from Fig5 that loss PSNR in layer gradually decays to near 0 with time in the proposed scheme, while there is no evident PSNR increase if no leaky prediction is employed. In addition, as shown in Fig. Fig5, the proposed scheme recovers the PSNR faster, since better layer quality has upgraded the overall frame quality by the proposed scheme, though the background slice is coded without leaky prediction. In Fig.6, continuous errors occur every 50 frames in the background slice transmission in 250 framesforeman. And the proposed scheme achieves gains, up to more than 2dB, in both layer and overall picture compared to that without leaky prediction. Since the leaky factor aims to get a tradeoff between robustness and efficiency, it would inevitably introduces losses in coding efficiency in error-free case due to two reasons: 1) The coded MB referring to the previous layer is not always likely to find a good match in the constrained ME, and this would cause larger residue, thus leading
PSNR(dB) 2 0 overall 0.9 overall 0.8 overall 0.5 0.9 0.8 0.5 Table 1. Bitrate varying with leaky factor α in foreman QCIF in error periodically every 50 frames (QP=28,12.5Hz) and bitrate increase compared to coding without leaky prediction α bitrate(kbps) increase(%) 1.0 62.96 0 0.9 83.77 33.05 0.8 93.03 47.76 0.5 116.48 85.01-16 0 2 4 6 8 10 12 14 16 18 20 0 Fig. 8. PSNR loss of the decoded layers and the whole pictures for different settings of leaky factor α in error case in foreman QCIF (QP=28,12.5Hz,Single, BK = 0) to a fair amount of bit increase. 2) It is required to transmit two sets of block mode and motion vectors for each MB within (coded with leaky prediction). And additional bitrate has to be consumed for the extra set of the side information. PSNR(dB) -16 BK 0 BK 1 BK 4 Fig.7 shows the RD curve of the proposed scheme, in both low bitrate case in foreman QCIF (Fig.7(a)) and high bitrate application in stefan CIF (Fig.7(b)). On average, 1dB PSNR loss in the whole frame and the is observed in slow motion sequence foreman, but the PSNR gap is narrowed with bit rate increase. Compared to the enhanced robustness in errorprone environments, the small loss in coding efficiency can be accepted. While in stefan sequence that featured fast motion, it is more difficult to find the match block by the constrained ME in than in slow motion video, making the RD performance even worse, up to 2dB PSNR loss in error-free case. 4.2. Influence of Parameters on Leaky Prediction Performance Leaky factor represents the evolvement of the layer reference in MC, hence it serves as a parameter to adjust the importance between coding efficiency and robustness. In this experiment, we fix the quantization steps QP and vary the leaky factor α. We use 250 frames foreman with continuous background error every 50 frames, and only slice is code with leaky prediction. It is seen in Fig.8 that the error resilience performance of the reconstructed layer and overall frame is closely related to the leaky factor. Generally, the smaller the leaky factor, the faster the errors decay, at a greater cost of bitrate, as illustrated in Tab.1. The number of MB using leaky prediction in scalable coding has a considerable effect on the proposed scheme. As -18 0 2 4 6 8 10 12 14 16 18 20 Fig. 9. PSNR loss of the decoded whole pictures for different settings of BK in error case in foreman QCIF (QP=28, 12.5Hz, Single, α = 0.9) is mentioned in Section.2.1, leaky prediction is not restricted to slice coding. Here we introduce a parameter BK to indicate the size of background slice involved in leaky prediction. Specifically, BK is the size of neighboring area, e.g. the extended 1 MB area encircling would use leaky prediction if BK = 1. Again, we fix QP and leaky factor in 250 frames foreman with continuous error every 50 frames, and vary BK from 0 (no MB in background slice using leaky prediction) to 4 (all the MB in background slice are using leaky prediction). Fig.9 shows the error robustness performance with BK for the overall frame only, in that PSNR curves of the layer will not have much noticeable difference with varying BK. Generally, larger BK corresponds to faster error recovery. Note that when BK = 4, i.e leaky prediction is applied to all macroblocks in one frame, PSNR decays almost to 0 for a periodical period of 4 seconds (error occurs every 50 frames and frames rate is 12.5Hz). The analysis of layer error mitigation in Section.3 is also applicable here, except to replace MB in slice with MB in either or background
38.5 38 37.5 38 37 36 anchor proposed PSNR(dB) 37 36.5 PSNR(dB) 35 36 35.5 anchor proposed 34 33 35 140 160 180 200 220 240 260 280 Bit Rate (kbps) (a) 32 400 450 500 550 600 650 700 750 800 850 900 Bit Rate(kbps) (b) Fig. 7. (a) RD comparison in error-free case in low bitrate sequence foreman QCIF. (b) RD comparison in error-free case in high bitrate sequence stefan CIF Table 2. Bitrate varying with BK in foreman QCIF in error case (QP=28,12.5Hz,α = 0.9, Single ) and bitrate increase compared to coding without leaky prediction BK bitrate(kbps) increase(%) 0 83.77 33.05 1 98.98 57.21 4 275.86 338.15 slice. However, the bitrate would increase substantially with BK, as is illustrated in Tab.2, because larger BK indicates that more background MB adjacent to region would be involved in leaky prediction coding. 5. CONCLUSION We have presented a novel scheme based on leaky prediction for scalable coding to enhance the quality of layer reconstruction. It makes a tradeoff between coding efficiency and error robustness in error-prone network. layer reference is also involved in MC loop for prediction in addition to the overall frame. Error propagation due to the prediction mismatch is effectively controlled by leaky prediction in the slice. Simulation results show that the error in layer of the video can be removed much faster from the decoded frame after a short period of time compared to scalable coding without leaky prediction, and the overall frame error degradation is accelerated as well. And a significant gain of up to 2dB in PSNR is reported in both layer and overall frame. Meanwhile it is observed the overall rate-distortion performance is close to that without leaky prediction in slow motion video, though a bit lower in high motion sequence. 6. REFERENCES [1] B. Girod, M. Kalman, Y. Liang, and R. Zhang, Advances in channel adaptive video streaming, in Proc. IEEE Int. Conf. Image Processing ICIP002, vol.1, Rochester, NY, Sep.2002, pp.9 [2] M. Flierl and B. Girod, Generalized B pictures and the draft H.264/AVC video-compression standard, IEEE Trans. on Circuits and Systems for Video Technology, vol.13, no.7, pp. 587-597, July 2003 [3] P. Lambert, W. De Neve, Y. Dhondt, R. Van de Walle, Flexible macroblock ordering in H.264/AVC, J.Visual Commun. Image Representation,vol.17, pp358-375, 2006 [4] Advanced video coding for generic audiovisual service,itu-t Recommendation H.264,Mar.2005 [5] Y. Wang and Q.F. Zhu, Error control and concealment for video communication: A review, Proc. IEEE, vol.86, pp.974-997, May 1998 [6] C. Chen, Error detection and concealment with an unsupervised MPEG2 video decoder, J. Visual Commu. Image Representation, vol.6, no.3, pp.26578, Sept. 1995 [7] B. Girod, and Niko Färber, Feedback-based error control for mobile video transmission, Proc. IEEE, vol.87, no.10, pp.1707-1723, Oct.1999 [8] N.S. Jayant and P. Noll, Digital Coding of Waveforms, Prentice-Hall, Englewood Cliffs, NJ, 1984 [9] S. Han, B. Girod, Robust and efficient scalable video coding with leaky prediction, IEEE 2002 International Conference on Image Processing, Rochester, New York, USA, Sepember, 2002 [10] http://ftp3.itu.ch/av-arch/jvt-site/reference software