Error Concealment Aware Rate Shapng for Wreless Vdeo Transport 1 Trsta Pe-chun Chen and Tsuhan Chen 2 Abstract Streamng of vdeo, whch s both source- and channel- coded, over wreless networks faces the challenge of tme-varyng packet loss rate and fluctuatng bandwdth. Rate shapng (RS) has been proposed to reduce the bt rate of a precoded vdeo btstream to adapt to the real-tme bandwdth varaton. In our earler work, rate shapng was extended to consder not only the bandwdth but also the packet loss rate varatons. Rate-dstorton optmzed rate adaptaton s performed on the precoded vdeo that s a scalable coded btstream protected by forward error correcton codes. In ths paper, we propose a rate shapng scheme that further takes nto account the error concealment (EC) method used at the recever. We refer to ths scheme as EC aware RS (ECARS). When performng ECARS, frst ECARS needs to know the beneft/gan of sendng each part of the precoded vdeo, as opposed to not sendng t but reconstructng t by EC. Then gven a certan packet loss probablty, the expected gan can be derved and be ncluded n the rate-dstorton optmzaton problem formulaton. Fnally ECARS performs rate-dstorton optmzaton to adapt the rate of the precoded vdeo. A two-stage rate-dstorton optmzaton approach s proposed to solve the ECARS rate -dstorton optmzaton problem. In addton to ECARS, the precodng process can be EC aware to prortze the precoded vdeo based on the gan. We present an example EC aware precodng process by means of macroblock prortzaton. Experment results of ECARS together wth EC aware precodng are shown to have excellent performance. Index Terms rate shapng, error concealment, rate-dstorton optmzaton, wreless vdeo 1 Work supported n part by Industral Technology Research Insttute. 2 The authors are wth Electrcal and Computer Engneerng, Carnege Mellon Unversty, Pttsburgh, PA 15213, U.S.A. Correspondng author: Prof. Tsuhan Chen, Electrcal and Computer Engneerng, Carnege Mellon Unversty, Pttsburgh, PA 15213, U.S.A. Tel: (412) 268-7536 Fax (412) 268-3890, E-mal: tsuhan@cmu.edu. 1
I. INTRODUCTION Due to the rapd growth of wreless communcatons, vdeo over wreless networks has ganed a lot of attenton. Challenges as to cope wth the tme-varyng error rate and fluctuatng bandwdth brng out the need of error reslent vdeo transport. Jont source-channel codng technques [1][2] are often appled to acheve error reslent vdeo transport wth onlne codng. However, jont source-channel codng technques are not sutable for streamng precoded vdeo. The precoded vdeo s both source- and channel- coded pror to transmsson. The network condtons are not known at the tme of codng. Rate shapng, whch was called dynamc rate shapng (DRS) n [3]-[7], was proposed to shape, that s, to reduce, the bt rate of the snglelayered pre source-coded (pre-compressed) vdeo, to meet the real tme bandwdth requrement. In [3]- [5], t was proposed to drop the dscrete cosne transform (DCT) coeffcents beyond the breakpont to reduce the bt rate of the pre-compressed vdeo; whle n [6][7], t was proposed to drop some blocks n a frame and reconstruct those that were dropped by nterpolaton at the recever, to reduce the bt rate of the pre-compressed vdeo. To protect the vdeo from transmsson errors n the wreless networks, source-coded vdeo btstream s often protected by forward error correcton (FEC) codes [8]. Redundant nformaton, known as party bts, s added to the orgnal source-coded bts. Party bts are ncluded n the precoded vdeo because FEC encodng at the tme of transmsson may not be feasble gven the capablty of the node that s transportng the vdeo. On the other hand, ths node should be able to perform rate shapng for both the source- and channel- coded btstream snce rate shapng has less complexty than full decodng. Ths node s able to perform full decodng f t wants to vew the content of the vdeo. Conventonal DRS dd not consder shapng for the party bts n addton to the source-codng bts. In our earler work, we extended rate shapng for transportng the precoded vdeo that s both pre 2
source- and channel- coded [9], whch we refer to as baselne rate shapng (BRS). The source codng n partcular refers to scalable vdeo codng as used by H.263 [10] and MPEG-4 [11]. By means of dscrete rate-dstorton (R-D) combnaton, BRS drops part of the precoded vdeo to acheve the best vdeo qualty. The part beng dropped can consst of bts from the scalable codng or the party bts from the FEC codng. In ths paper, we propose a rate shapng scheme that further takes nto account the error concealment (EC) method used at the recever. We refer to ths scheme as EC aware RS (ECARS). Related work that utlzed EC nformaton for rate shapng on pre source- coded btstream only can be found n [6][7]. When performng ECARS, frst ECARS needs to know the beneft/gan of sendng each part of the precoded vdeo, as opposed to not sendng t but reconstructng t by EC. The gan of sendng some part of the precoded vdeo s large f the EC method used at the recever cannot reconstruct ths part very well. Such gan wll be dfferent f the EC method consdered s dfferent. Gan nformaton can ether be computed at the tme of transmsson or be embedded n the btstream. Then gven a certan packet loss probablty, the expected gan can be derved and be ncluded n the R-D optmzaton problem formulaton. Fnally ECARS performs R-D optmzaton to adapt the rate of the precoded vdeo. A two-stage R-D optmzaton approach s proposed to solve the ECARS R-D optmzaton problem. Pror work on R-D optmzaton ncludes [12]-[15], where [12]-[14] solved the lnear programmng/nteger programmng problem by Lagrangan multpler wth prunng and teratve bsecton, and [15] used a hll clmbng based approach called senstvty adaptaton (SA) algorthm. The proposed two-stage R-D optmzaton ams for both effcency and optmalty by usng the model-based hyper-surface and the hll clmbng based refnement. In addton to ECARS, the precodng process can be EC aware to prortze the precoded vdeo based on the gan. We present an example EC aware precodng process by means of macroblock (MB) prortzaton. A MB n a frame s ranked accordng to ts gan, whch depends on how well ths MB can 3
be reconstructed by the EC method used at the recever. The gan of sendng a MB s large f the EC method used at the recever cannot reconstruct ths MB very well. Ths paper s organzed as follows. In Secton II, we ntroduce baselne rate shapng (BRS) and error concealment (EC) as the background. In Secton III, error concealment aware rate shapng (ECARS) s proposed. Gven any precoded vdeo, ECARS frst evaluates the gan consderng a partcular EC method used at the recever. ECARS then performs a two-stage R-D optmzaton for rate adaptaton under the current network condton, n terms of packet loss rate and bandwdth. In addton, we also ntroduce EC aware precodng where a MB prortzaton scheme s presented. In Secton IV, experment results of ECARS together wth EC aware precodng are shown. Concludng remarks are gven n Secton V. II. BACKGROUND We wll gve bref descrptons of baselne rate shapng (BRS) and error concealment (EC) n ths secton. BRS provdes a smple llustraton of what s nvolved n rate shapng for pre source- and channel- coded vdeo. In addton, snce the proposed ECARS takes nto account error concealment for rate shapng, we also descrbe brefly error concealment technques that may be used at the recever. A. Baselne Rate Shapng (BRS) There are three stages to transmt the vdeo from the sender to the recever: () precodng, () streamng wth BRS, and () decodng, as shown from Fgure 1 to Fgure 3. 4
Vdeo Scalable encoder enhancement layer btstream Base layer btstream FEC encoder FEC encoder Precoded Vdeo btstream Fgure 1. System dagram of the precodng process: scalable encodng followed by FEC encodng network condtons Precoded vdeo Baselne Baselne rate rate shaper shaper (BRS) (BRS) Wreless Network Fgure 2. Transport of the precoded vdeo wth BRS Wreless Network Shaped vdeo btstream FEC decoder Scalable decoder Reconstructed vdeo Fgure 3. System dagram of the decodng process: FEC decodng followed by scalable decodng BRS reduces the bt rate of each decson unt of the precoded vdeo before t sends the precoded vdeo to the wreless network. A decson unt can be a frame, a macroblock, etc., dependng on the granularty of the decson. We use a frame as the decson unt heren. Let us consder the case n whch the vdeo sequence s scalable coded nto two layers: one base layer and one enhancement layer. These two layers are FEC coded wth unequal packet loss protecton (UPP) capabltes. Therefore, there are four segments n the precoded vdeo. The frst segment conssts of the bts of the base layer vdeo btstream (upper left segment of Fgure 4 (a)). The second segment conssts of the bts of the enhancement layer vdeo btstream (upper rght segment of Fgure 4 (a)). The thrd segment conssts of the party bts for the base layer vdeo btstream (lower left segment of Fgure 4 (a)). The fourth segment conssts of the party bts for the enhancement layer vdeo btstream (lower rght segment of Fgure 4 5
(a)). BRS decdes a subset of the four segments to send. When the channel has abundant bandwdth, BRS wll send wth the confguraton shown n Fgure 4 (a). When the bandwdth s reduced, the second confguraton shown n Fgure 4 (b) s chosen. When the bandwdth s reduced even more, ether Fgure 4 (c) or Fgure 4 (d) wll be chosen dependng on the wreless network condton. A rule of thumb s to choose party bts to send nstead of bts of the enhancement layer when the packet loss rate s hgh. In the extreme case where the bandwdth s so lmted, none of the segments wll be chosen to be sent as shown n Fgure 4 (f). Interested readers can read more from [9], whch conssts of BRS by mode decson that we just descrbe and the dscrete R-D combnaton. (a) (b) (c) (d) (e) (f) Fgure 4. Sx dfferent combnatons of subset of the four segments B. Error Concealment (EC) Error concealment reles on some a pror knowledge to reconstruct the lost vdeo content. Such a pror knowledge can come from spatal or temporal neghbors. For example, we can assume that the pxel values are smooth across the boundary of the lost and retaned regons. To recover lost data wth the smoothness assumpton, nterpolaton or optmzaton based on certan objectve functons are often used. Fgure 5 and Fgure 6 show corrupted frames and the correspondng reconstructed frames. The black regons n Fgure 5 (a) and Fgure 6 (a) ndcate losses of the vdeo data. Fgure 5 shows an error concealment method usng spatal nterpolaton from the neghborng pxels. Fgure 6 shows an error concealment method usng temporal nterpolaton. That s, f some pxel values are lost, the decoder 6
copes the pxel values from the prevous frame at the correspondng locatons to the current frame. The error concealment method us ng temporal nterpolaton can be extended to copyng the pxel values from the prevous frame at the moton-compensated locatons. The moton vectors used for moton compensaton ether are assumed error-free or can be estmated at the decoder [16][17]. We use the smple temporal nterpolaton method n ths paper. Future extenson ncludes usng moton-compensated temporal nterpolaton, or more sophstcated error concealment methods as mentoned n [18]. (a) (b) Fgure 5. Error concealment example by spatal nterpolaton: (a) the corrupted frame wthout error concealment, and (b) the reconstructed frame wth error concealment (a) (b) Fgure 6. Error concealment example by temporal nterpolaton: (a) the corrupted frame wthout error concealment, and (b) the reconstructed frame wth error concealment 7
III. ERROR CONCEALMENT AWARE RATE SHAPING (ECARS) In ths secton, we wll start from descrbng the wreless vdeo transport system, ncludng precodng, streamng wth rate shapng, and decodng. We then propose the EC aware RS scheme (ECARS), that frst evaluates the gans, whch we wll defne formally, consderng a partcular EC method used at the recever, then performs the two-stage R-D optmzaton. In addton, f the system allows for EC aware precodng, ECARS can take advantage of that. We wll present an EC aware precodng process by means of MB prortzaton. A. Wreless Vdeo Transport System There are three stages to transmt the vdeo from the sender to the recever n a wreless vdeo transport system: () precodng, () streamng wth rate shapng, and () decodng, as shown from Fgure 7 to Fgure 9. In the precodng process (shown n Fgure 7), vdeo s encoded by both the source encoder and the FEC encoder. The precodng process s done before the tme of delvery. The precodng process may be aware of the EC used at the recever, whch we wll descrbe later. Notce that n ths paper, the precoded vdeo for ECARS s pre source-coded wth a sngle layer. In the streamng stage (shown n Fgure 8), ECARS takes the network condtons as the bandwdth and the packet loss rate nto account to acheve the best vdeo qualty. The decodng process (shown n Fgure 9) conssts of FEC decodng followed by scalable decodng. Precodng process (can be EC aware) Vdeo Source encoder FEC encoder Precoded Vdeo btstream Fgure 7. System dagram of the precodng process: source encodng (whch can be EC aware) followed by FEC encodng 8
network condtons Precoded vdeo EC EC aware aware RS RS (ECARS) Wreless Network Fgure 8. Transport of the precoded vdeo wth ECARS Wreless Network Shaped vdeo btstream FEC decoder Source decoder Reconstructed vdeo Fgure 9. System dagram of the decodng process: FEC decodng followed by source decodng B. R-D Optmzaton for ECARS Gven the precoded vdeo, whch s both source- and channel- coded, ECARS wll perform bandwdth adaptaton for streamng. We start from a smple example as an extenson to BRS then gve a more general ECARS. Let us consder that the precoded vdeo conssts of two layers of vdeo btstream, namely, the base layer and the enhancement layer. Each layer s protected by party bts from the FEC codng. The settng s shown earler n Fgure 4 (a). The rate shaper s extended to gve a fner decson on how many symbols 3 to send (or how many symbols to drop) for each layer, nstead of decdng whch segment(s) to drop as suggested by BRS. Snce the rate shaper s aware of the EC method used at the recever, t can evaluate how much dstorton decrease t can get n f the rate shaper decdes to send a certan amount of symbols for each layer. In general, the base layer can be reconstructed well wth error concealment snce the base layer conssts of coarse nformaton of the vdeo that can be easly reconstructed. On the other 3 Symbols are used nstead of bts snce the FEC codes use a symbol as the encodng/decodng unt. In ths paper, we use 14 bts to form one symbol. The selecton of symbol sze n bts depends on the user. 9
hand, the enhancement layer, whch conssts of fne detals of the vdeo, cannot be easly reconstructed. More dstorton decrease could be obtaned f the rate shaper decdes to send the enhancement layer vdeo. In ths case, the EC aware rate shaper would assgn a hgher gan (dstorton decrease) on sendng symbols from the enhancement layer than the symbols from the base layer. Note agan that n ths paper, the precoded vdeo for ECARS s pre source-coded wth a sngle layer. Ths sngle layer of vdeo btstream wll be arranged nto sublayers, whch we wll defne shortly. The sublayers shall not be confused wth the two-layered example gven n the last paragraph for llustraton purpose only. Havng understood how the gan of sendng some part of the precoded vdeo s determned consderng the EC used at the recever, we can now ntroduce a more general ECARS. Suppose ECARS s gven the precoded vdeo consstng of several sublayers. The sublayers are usually arranged n a way that the lower sublayers are more mportant n reconstructng the vdeo qualty than the hgher sublayers are. That s, lower sublayers are assocated wth larger sublayer gans G s, where s the sublayer ndex; and hgher sublayers are assocated wth smaller sublayer gans G s. We wll descrbe n more detal n Secton III. C such a precodng process and defnton of the sublayer gans. As shown n Fgure 10 (a), the upper porton of each strpe conssts of the symbols from source codng, and the lower porton of each strpe conssts of the symbols from channel codng. The darken bars n Fgure 10 (b) represent the symbols to be sent by ECARS. Sublayer 1 2 3 h Sublayer 1 2 3 h 10
(a) (b) Fgure 10. (a) Precoded vdeo n sublayers and (b) ECARS decson on whch symbols to send The problem formulaton for ECARS s as follows. The total gan s ncreased (or the total dstorton s decreased) as more sublayers are correctly decoded. Wth Sublayer 1 correctly decoded, the total gan s ncreased by G 1 (accumulated gan s G 1 ); wth Sublayer 2 correctly decoded, the total gan s ncreased further by G 2 (accumulated gan s G 1 + G2 ); and so on. Note that G of Sublayer s calculated gven the EC method used at the recever, thus EC aware. G of Sublayer s dfferent for every frame. Snce the precoded vdeo s transmtted over error prone wreless networks, sublayers are subject to loss and have certan recovery rates gven a partcular rate shapng decson. The expected accumulated gan s then: G = h = 1 G v (1) f each sublayer can be decoded ndependently 4. v s the recovery rate of Sublayer that s a functon of r as shown later n (2). Usng Reed-Solomon codes as the channel codes n ths paper, Sublayer s recoverable (or successfully decodable) f the number of erasures resultng from the lossy transmsson s no more than r k. k s the message (symbols from the source codng) sze of Sublayer and r s the number of symbols selected to be sent n Sublayer. Wth Reed-Solomon codes used, r k wth the excepton of the last sublayer (not necessary the Sublayer h, can be the sublayer before that); and the whole sublayer s consdered lost f the number of erasures s beyond the error-correcton 4 If Sublayer can be decoded only f Sublayer 1 s decoded correctly, (1) can be modfed to G = h = 1 j = 1. G v j 11
capablty r k. Thus, the recovery rate v s the summaton of the probabltes that no loss occur, one erasure occurs, and so on untl v = r k erasures occur. r k l= 0 r l l ( e ) ( e ) m r l 1, = 1 ~ h (2) m where h s the number of sublayers of ths frame n total and e m s the symbol loss rate. The symbol loss rate can be derved from the packet loss rate as ( ) s e m e p m = 1 1, where s s the packet sze and m s the symbol sze n bts. By choosng dfferent combnatons of the number of symbols for each sublayer, the expected accumulated gan wll be dfferent. The rate shapng problem can be formulated as follows: maxmze G = h = 1 G v subject to h r B =1 (3) where B s the bandwdth constrant ths frame has to satsfy. To solve ths problem, we propose a new two-stage R-D optmzaton approach. The two-stage R-D optmzaton frst fnds the near-optmal soluton globally. The near-optmal global soluton s then refned by a hll clmbng approach. Pror work on R-D optmzaton ncludes [12]-[15]. The proposed two-stage R-D optmzaton s dfferent from [12]-[15] n two folds. Frst, the model-based Stage 1 allows us to examne fewer samples from all the operatonal R-D states. Second, the proposed dstorton measure (or expected accumulated gan n the termnology of ths paper) accounts for the effects of packet loss as well as the channel codes by means of recovery rates. 12
1) Two-stage R-D Optmzaton: Stage 1 We can see from (1) and (2) that the expected accumulated gan G s related to r = [ r r L ] mplctly through the recovery rates v = [ v v L ] 1 2 v h. We can nstead fnd a model-based hypersurface that explctly relates r and G. The model parameters can be traned from a set of tranng data ( r,g), where r values are chosen by the user and G values can be computed by (1) and (2). The optmal soluton s the feasble soluton wthn the ntersecton of the hyper-surface and the bandwdth constrant as llustrated n Fgure 11. A complex model, wth a lot of parameters, can be used to descrbe as close as possble the true dstrbuton of the R-D states. The soluton obtaned from the ntersecton wll be as close to optmal as possble. However, the number of ( r,g) pars needed to tran the modelbased hyper-surface ncreases wth the number of parameters. 1 2 r h G r 1 r 1 +r 2 =B r 2 Fgure 11. Intersecton of the model-based hyper-surface (dark surface) and the bandwdth constrant (gray plane), llustrated wth h = 2 follows: In ths paper, we use a quadratc equaton to descrbe the relaton between r and G as h h G = 2 a r + bjr rj + cr + d = 1, j= 1, j = 1 h (4) 13
In ths paper, the model parameters a, b j, c, and d are traned dfferently for each frame. They can be solved by surface fttng wth a set of tranng data ( r,g) obtaned from (1) and (2). For example, the parameters can be computed by: a 's bj's = c 's d T ( R R) 1 R T G G M G 1 2 Ξ (5) where the left super ndex of G s the ndex of the tranng data, R s a matrx consstng Ξ rows of ( 2 's, r r 's, r 's, 1) r. The complexty of computng a s, j b j s, c s, and d relates to the number 2 of parameters h + h + 1 and the number of tranng data Ξ, usng (5). Note that the number of tranng 2 data Ξ s n general much greater than the number of parameters h + h + 1. Thus, a more complex 3 2 model, such as a thrd-order model wth h + h + h + 1 parameters, wll not be sutable snce t requres much more tranng data. In addton, Second-order Taylor expanson can approxmate ncely n general every functon. (4) can be seen as a second-order approxmaton to (1)(2). To reduce the computaton complexty n realty, we can also choose a smaller h. Wth (4), the near-optmal soluton can be obtaned by Lagrangan multpler as follows. J h h h = + + + + h 2 ar bjr rj cr d λ r B (6) = 1, j= 1, j = 1 = 1 J By = 0, we get: r r h 1 = bjr 2a j= 1, j j + c + λ (7) where λ s: 14
h h 1 2B + bjrj + c = 1 a j= 1, j λ = (8) h 1 a The near-optmal soluton can be solved recursvely usng (7) and (8), startng from the ntal condton = 1 that all sublayers are allocated wth equal number of symbols, B r = r = L = r = 2 h h 1. 2) Two-stage R-D Optmzaton: Stage 2 Stage 1 of the two-stage R-D optmzaton gves a near-optmal soluton. The soluton can be refned by a hll-clmbng based approach (Fgure 12). The soluton from Stage 1 s perturbed n order to yeld a larger expected accumulated gan. The process can be terated untl the soluton reaches a stoppng crteron such as the convergence. Whle (stop == false) z = r for all =1~h For (j=1; j<=h; j++) For (k=1; k<=h; k++) z k = z k + delta for k==j //Increase sublayer j z k = z k - delta/(h - 1) for k!=j //Decrease others End - for Evaluate G j by equatons (1) and (2) End - for Fnd the j* wth the largest G j *. For (=1; <=h; ++) r = r + delta for ==j* r = r - delta/(h - 1) for!=j* End - for Calculate the stop crteron. End - whle Fgure 12. Pseudocodes of hll-clmbng algorthm 15
C. Error Concealment Aware Precodng In addton to ECARS, the precodng process can be EC aware to prortze the precoded vdeo based on the gan. We present an example EC aware precodng process by means of macroblock (MB) prortzaton. A MB n a frame s ranked accordng to ts gan, whch depends on how well ths MB can be reconstructed by the EC method used at the recever. The gan of sendng a MB s large f the EC method used at the recever cannot reconstruct ths MB very well. Let us consder that a smple temporal nterpolaton based EC method s adopted. Fgure 13 provdes us wth an llustraton of EC aware MB prortzaton. If MB ( 11, ) s lost n Frame n, t cannot be well reconstructed by MB ( 1, ) 1 from Frame 1 n, t can be well reconstructed by MB (,3) wth hgher prorty than MB ( 0,3). n. On the other hand, f MB (,3) 0 from Frame 1 0 s lost n Frame n. Therefore, we should rank MB ( 11, ) We can use square sum of the pxel dfferences between the orgnal MB and the ECreconstructed MB as the measure for prorty. The larger the square sum s, the larger the gan for ths MB s, thus, the hgher the prorty of ths MB s. Assumng that the neghborng MB of the MB consdered are decoded wthout errors, the MB gan ( c ju p ju s ju ) g j s defned as follows: = 255 2 g j, =1 ~ number of MB n a frame u= 0 j (9) where u 5 s the coeffcent ndex n a MB, c ju s the coeffcent of the EC-reconstructed MB, p ju s the predcton value of ths MB, and s ju s the resdue value of ths MB. ju s ju wthout any transmsson error or rate adaptaton by rate shapng. c ( p + s ) ju ju p + s the deal value s to see how far the ju 5 We consder only the Y components n the MB wthout loss of generalty. Thus, there are four 8 8 blocks or 256 coeffcents nsde. 16
EC value s from the deal value. The assumpton that the neghborng MB are decoded wthout errors s vald f the packet losses are not too bursty. (0,0) (0,1) (0,2) (1,0) (1,1) (1,2) (2,0) (2,1) (2,2) (0,3) (1,3) (2,3) (a) (b) (c) Fgure 13. (a) Frame n 1, (b) Frame n, and (c) MB ndces. EC aware MB prortzaton MB (1,1) has hgher prorty than MB (0, 3) An observaton to make s that the conventonal vdeo codng can be consdered as a specal case of the proposed EC aware MB prortzaton. Let us consder the case where no moton vector s used n vdeo codng. The MB wth large resdues s encoded and transmtted, whle the MB wth small resdues does not need to be transmtted snce the small resdues wll become zero after quantzaton. Ths case translates to the case of EC aware MB prortzaton usng temporal nterpolaton wth zero moton vectors. Let us consder another case where moton vectors are ncluded n vdeo codng. Ths then translates to the case of EC aware MB prortzaton usng temporal nterpolaton wth moton vectors. We can see that the proposed EC aware MB prortzaton s more general snce t s not lmted to any specfc error concealment method. The source-coded btstream wth EC aware MB prortzaton can be appended wth party bts from the FEC codng. Frst, the bts of the hghest prorty MB s placed followed by the bts of the second hghest prorty MB and so on, as shown n Fgure 14 (a). To label the MB after the MB are 17
ordered by ther prortes, 446 bytes of complementary nformaton of the MB labels are needed f the vdeo s n common ntermedate format (CIF). The bts are then dvded nto sublayers as shown n Fgure 14 (b). Sublayer + 1 has more bts than Sublayer snce we want to acheve UPP for the sublayers when appended wth the party bts. For example, we can let Sublayer 1 conssts of bts from the frst 10 hghest prorty MB, Sublayer 2 conssts of bts from the followng 20 hghest prorty MB, and so on. Each sublayer s then appended wth party bts from the FEC codng as shown n Fgure 14 (c). MB prortzed btstream bts of MB (1,1) bts of MB (0,1) Sublayer 1 2 3 h Sublayer 1 2 3 h bts of MB (0,3) (a) (b) (c) Fgure 14. Precoded vdeo: (a) MB prortzed btstream, (b) MB prortzed btstream n sublayers, and (c) FEC coded MB prortzed btstream Also, wth the MB gan defned, we can defne the sublayer gan correspondngly as: j G = g j, = 1 ~ number of sublayers n a frame { ndces of MB that belongtosublayer } (10) Note agan that ECARS can perform rate adaptaton wth or wthout EC aware precodng as long as the precoded vdeo s provded wth sublayer gans. To summarze, the proposed ECARS wth EC aware precodng utlzes the MB gans consderng the EC method used at the recever. The expected accumulated gan used n the later R-D 18
Order of droppng optmzaton s not only based on the MB gans but also on the current network condton. A two-stage R-D optmzaton approach s then proposed for fndng the optmal soluton. IV. EXPERIMENT In the experment, we wll show results of the proposed ECARS together wth EC aware precodng, compared wth the naïve rate shapng method unequal error protecton rate shapng (UPPRS) descrbed n Fgure 15. UPPRS wll drop from the bottom f the bandwdth s not enough. In that, UPP can be acheved snce more party symbols are sent for Sublayer than Sublayer + 1. Sublayer 1 2 3 h Fgure 15. UPPRS llustraton Wreless networks are generally wth tme-varyng packet loss rate and fluctuatng bandwdth. The packet loss rate and bandwdth vary at each tme nterval. The tme nterval of our smulaton s the frame nterval (33 ms for a frame rate of 30 frames/sec). We smulate random bandwdth fluctuaton and use a two-state Markov-chan [19][20] (Fgure 16) to smulate the bursty bt errors. Example traces of smulated bandwdth and packet loss rate are shown n Fgure 17. In realty, through standards such as the real-tme control protocol (RTCP, part of the real-tme transport protocol (RTP)) [21], rate shaper can obtan network condton nformaton. The delay of such network condton nformaton s n general less than a frame nterval gven the one-way transmsson tme descrbed n [22]. 19
1-p 1-q p Good q Bad Fgure 16. Two-state Markov chan for bt error smulaton x 10 5 bandwdth (bytes/sec) 4.5 4 3.5 3 2.5 2 1.5 1 0.5 packet loss rate 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 50 100 150 200 250 300 0 50 100 150 200 250 300 tme ndex tme ndex (a) (b) Fgure 17. Traces of: (a) Bandwdth and (b) packet loss rate The test vdeo sequences are akyo, foreman, and stefan n common ntermedate format (CIF) (Fgure 18 (a)-(c)). We use H.263 [10] for vdeo encodng. Results n the followng are shown for the lumnance Y components only. 20
(a) (b) (c) Fgure 18. Test vdeo sequences n CIF: (a) akyo, (b) foreman, and (c) stefan Fgure 19 to Fgure 21 show the EC aware precodng by MB prortzaton. A MB s more mportant than the others are, f ts square sum of the pxel dfferences between the orgnal MB and the EC-reconstructed MB (that s the MB of the prevous frame n ths paper) s larger. The brghter the MB s, the larger the MB gan s, and hence the hgher the MB prorty s. In Fgure 19, the only scene varaton s from the anchor, mostly n the head and mouth regons. In Fgure 20, most of the scene varatons are from the head of the foreman. In Fgure 21, the scene vara tons are from the movements of the tenns player and the camera moves. EC-reconstructed MB dffers more from the orgnal MB n those regons wth more scene varatons. Thus, the MB n those regons s shown wth brghter ntensty. (a) (b) (c) Fgure 19. EC aware MB prortzaton of Sequence akyo n (a) Frame 2, (b) Frame 32, and (c) Frame 122 21
(a) (b) (c) Fgure 20. EC aware MB prortzaton of Sequence foreman n (a) Frame 2, (b) Frame 32, and (c) Frame 122 (a) (b) (c) Fgure 21. EC aware MB prortzaton of Sequence stefan n (a) Frame 2, (b) Frame 32, and (c) Frame 122 Frame by frame PSNR results for Sequence akyo, foreman and stefan are shown n Fgure 22, Fgure 23, and Fgure 24, respectvely. The overall PSNR performance for all three test sequences s shown n Fgure 25. We can see that the proposed ECARS performs better than UPPRS. The mprovement of ECARS over UPPRS s the most sgnfcant n Sequence stefan followed by Sequence foreman and akyo. Sequence stefan s dffcult to be reconstructed well by error concealment f the vdeo data s lost durng the transmsson. It s more crucal to send the rght 22
combnaton of symbols that s aware of the EC method at the recever. Therefore, the performance mprovement of ECARS over UPPRS s more promnent. akyo: Y akyo: Y 38.8 38.8 38.6 38.6 PSNR (db) 38.4 38.2 UPPRS ECARS PSNR (db) 38.4 38.2 UPPRS ECARS 38.0 38.0 37.8 0 50 100 150 200 250 300 frame number 37.8 150 160 170 180 190 200 frame number (a) (b) Fgure 22. Frame by frame PSNR of UPPRS and ECARS wth Sequence akyo : (a) result from Frame 1 to Frame 300, (b) zoomed result from Frame 150 to Frame 200 foreman: Y foreman: Y 39 39 PSNR (db) 38 37 36 35 34 33 32 UPPRS ECARS PSNR (db) 37 35 33 31 UPPRS ECARS 31 0 50 100 150 200 250 300 29 150 160 170 180 190 200 frame number frame number (a) (b) Fgure 23. Frame by frame PSNR of UPPRS and ECARS wth Sequence foreman : (a) result from Frame 1 to Frame 300, (b) zoomed result from Frame 150 to Frame 200 23
stefan: Y stefan: Y 36 36 34 34 PSNR (db) 32 30 UPPRS ECARS PSNR (db) 32 30 UPPRS ECARS 28 0 50 100 150 200 250 300 28 150 160 170 180 190 200 frame number frame number (a) (b) Fgure 24. Frame by frame PSNR of UPPRS and ECARS wth Sequence stefan : (a) result from Frame 1 to Frame 300, (b) zoomed result from Frame 150 to Frame 200 38 38.49 38.54 PSNR (db) 35 32 34 34.61 30.67 32.34 UPPRS ECARS 29 akyo foreman stefan sequence Fgure 25. Overall PSNR of UPPRS and FGRS wth sequences akyo, foreman, and stefan Some sample frames are shown n Fgure 26, Fgure 27, and Fgure 28 for the three test sequences. These three examples show the cases where UPPRS does not perform as well as ECARS. In Fgure 26, UPPRS does not protect the MB n the eye regons well enough as ECARS. The MB n the eye regons are thus corrupted. Error concealment reconstructs the corrupted MB wth the pxel values of the prevous frame. The current frame has the eyes closed whle the prevous frame has the eyes open. On the other hand, ECARS protects the MB n the eye regons well enough and thus does not 24
result n corrupted MB n the eye regons. Smlarly, Fgure 27 and Fgure 28 show that the MB n the hat and body regons, respectvely, are protected better by ECARS than UPPRS. (a) (b) Fgure 26. Example decoded frame, Frame 5, of Sequence akyo wth (a) UPPRS and (b) ECARS (a) (b) Fgure 27. Example decoded frame, Frame 150, of Sequence foreman wth (a) UPPRS and (b) ECARS 25
(a) (b) Fgure 28. Example decoded frame, Frame 181, of Sequence stefan wth (a) UPPRS and (b) ECARS To examne how ECARS outperforms UPPRS, we look at the MB recovery rates of all the MB n three sample frames, Frame 2, Frame 32, and Frame 122. Wth the Reed-Solomon codes used n ths paper, the MB recovery rates can be computed gven the R-D optmzaton result r = [ r r L ] of the frame examned. We can verfy the valdty of the proposed rate shapng algorthm f the MB that s harder to be reconstructed well by error concealment has hgher recovery rate. Fgure 29 and Fgure 30 show the MB recovery rates of Sequence akyo, Fgure 31 and Fgure 32 show the MB recovery rates of Sequence Sequence foreman, and Fgure 33 and Fgure 34 show the MB recovery rates of Sequence stefan. Fgure 29, Fgure 31, Fgure 33 are the results by UPPRS whle Fgure 30, Fgure 32, and Fgure 34 are the results by ECARS. The brghter the MB s, the hgher the probablty t can be receved wthout errors. The recovery rate s determned by the vdeo transport scheme, that s, ether UPPRS or ECARS. We can see that Fgure 30 resembles Fgure 19 more than Fgure 29 does. Smlarly, Fgure 32 resembles Fgure 20 more than Fgure 31 does; and Fgure 34 resembles Fgure 21 more than Fgure 33 does. Wth ECARS, the MB that s wth hgher prorty ndeed gets hgher recovery rate. 1 2 r h 26
(a) (b) (c) Fgure 29. MB loss recovery rates of Sequence akyo n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng UPPRS (d) (e) (f) Fgure 30. MB loss recovery rates of Sequence akyo n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng ECARS 27
Fgure 31. MB loss recovery rates of Sequence foreman n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng UPPRS Fgure 32. MB loss recovery rates of Sequence foreman n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng ECARS 28
Fgure 33. MB loss recovery rates of Sequence stefan n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng UPPRS Fgure 34. MB loss recovery rates of Sequence stefan n (a) Frame 2, (b) Frame 32, and (c) Frame 122 usng ECARS V. CONCLUSION We proposed n ths paper error concealment aware rate shapng (ECARS) for vdeo transport over wreless networks. ECARS s appled to pre source- and channel- coded vdeo. ECARS frst evaluates the gan of sendng the MB of the precoded vdeo, as opposed to not sendng t but reconstructng t by EC. Then gven a certan packet loss rate, the expected accumulated gan can be derved and be ncluded 29
n the R-D optmzaton problem formulaton. Fnally, ECARS performs R-D optmzaton by the proposed two-stage R-D optmzaton approach. The proposed two-stage R-D optmzaton approach frst obtans the near-optmal soluton by fndng the ntersecton of the model-based hyper-surface and the bandwdth constrant, and refnes the soluton from Stage 1 by a hll-clmbng based approach. Furthermore, the precodng process can be EC aware to prortze the precoded vdeo based on the MB gans. The proposed ECARS outperforms the naïve UPPRS approach n the experment. The expected accumulated gan dscussed n ths paper s defned wthn each frame. All the frames are ntra-coded and the decson made by the rate shaper wll not affect the frames that follow. Future work ncludes extendng ECARS for vdeo wth frame dependency, e.g. nter-coded vdeo. Some dscussons can be found n [23]. Feedback nformaton, such as whch MB s corrupted and the mean of the corrupted MB, s used by ECARS wth frame dependency consderaton. The way the MB are grouped nto sublayers n ths paper s fxed and s not part of the ECARS R-D optmzaton, snce how MB are grouped should be consdered n the precodng process but not n the rate shapng stage. In the future, we can consder R-D optmzaton on the way MB are grouped nto sublayers (that s, the number of source-coded symbols that go to each sublayer) gven the rate shapng problem s solved. VI. REFERENCE [1] G. Cheung and A. Zakhor, Bt Allocaton for Jont Source/Channel Codng of Scalable Vdeo, IEEE Transactons on Image Processng, 9(3), March 2000. [2] L. P. Kond, F. Ishtaq, and A. K. Katsaggelos, Jont Source-Channel Codng for Moton-Compensated DCT-based SNR Scalable Vdeo, IEEE Transactons on Image Processng, 11(9), September 2002. [3] A. Eleftherads and D. Anastassou, Meetng Arbtrary QoS Constrants usng Dynamc Rate Shapng of Coded Dgtal Vdeo, NOSSDAV 1995, pp. 96-106, Durham, New Hamp shre, Aprl 1995. [4] A. Eleftherads and D. Anastassou, Constraned and General Dynamc Rate Shapng of Compressed Dgtal Vdeo, ICIP 1995, vol. 3, pp. 396-399, Washngton D.C., October 1995. [5] S. Jacobs and A. Eleftherads, Streamng Vdeo Usng Dynamc Rate Shapng and TCP Congeston Control, Journal of Vsual Communcaton and Image Representaton, 9(3), 1998, pp. 211-222. 30
[6] W. Zeng and B. Lu, Rate Shapng by Block Droppng for Transmsson of MPEG-precoded Vdeo over Channels of Dynamc Bandwdth, ACM Multmeda 96, Boston, MA, U.S.A, 1996. [7] W. Zeng and B. Lu, Geometrc -Structure-Based Error Concealment wth Novel Applcatons n Block- Based Low-Bt -Rate Codng, IEEE Transactons on Crcuts and Systems for Vdeo Technology, 9(4), June 1999, pp. 648-665. [8] S. Wcker, Error Control Systems for Dgtal Communcaton and Storage, Prentce-Hall, 1995. [9] T. P.-C. Chen and T. Chen, Adaptve Jont Source-Channel Codng usng Rate Shapng, ICASSP 2002, Orlando, FL, U.S.A., May 2002. [10] D. S. Turaga and T. Chen, Fundamentals of Vdeo Compresson: H.263 as an Example, n Compressed Vdeo over Networks, edted by M.-T. Sun and A. R. Rebman, Marcel Dekker, Inc., 2001. [11] Moton Pctures Experts Group, "Overvew of the MPEG-4 Standard", ISO/IEC JTC1/SC29/WG11 N2459, 1998. [12] Y. Shoham and A. Gersho, Effcent Bt Allocaton for an Arbtrary Set of Quantzers, IEEE Transactons on Acoustc, Speech, and Sgnal Processng, 36(9), September 1988, pp. 1445-1453. [13] K. Ramchandran, A. Ortega, and M. Vetterl, Bt Allocaton for Dependent Quantzaton wth Applcatons to Multresoluton and MPEG Vdeo Coders, IEEE Transactons on Image Processng, 3(5), September 1994, pp. 533-545. [14] A. Ortega and K. Ramchandran, Rate-Dstorton Methods for Image and Vdeo Compresson. IEEE Sgnal Processng Magazne, 15(6), November 1998, pp. 23-50. [15] P. A. Chou and Z. Mao, Rate-Dstorton Optmzed Streamng of Packetzed Meda, submtted to IEEE Transactons on Multmeda, February 2001. [16] W. M. Lam, A. Rebman, and B. Lu, Recovery of Lost or Erroneously Receved Moton Vectors, ICASSP 1993, vol. 5, pp. 417-420. [17] M. E. Al-Mualla, N. Canagarajah, D. R. Bull, Multple -reference temporal error concealment, ISCAS 2001, vol. 5, pp. 149-152. [18] Trsta Pe-chun Chen and Tsuhan Chen, "Second-Generaton Error Concealment for Vdeo Transport over Error Prone Channels", Wreless Communcatons and Moble Computng, Specal Issue on Multmeda over Moble IP, October 2002. [19] F. Alajaj and T. Fuja, A Communcaton Channel Modeled on Contagon, IEEE Transactons on Informaton Theory, 40(6), pp. 2035-2041, 1994. [20] M. Yajnk, S. Moon, J. Kurose, D. Towsley, Measurement and Modelng of the Temporal Dependence n Packet Loss, INFOCOM 1999, pp. 345-52, March 1999. [21] H. Schulzrnne, S. Casner, R. Frederck, and V. Jacobson: RTP: A Transport Protocol for Real-Tme Applcatons, RFC1889, Jan. 1996. ftp://ftp.s.edu/n-notes/rfc1990.txt. [22] ITU-T Recommendaton G. 114, One-Way Transmsson Tme, May 2000. [23] Trsta Pe-chun Chen and Tsuhan Chen, "Rate Shapng for Vdeo wth Frame Dependency", ICME 2003, Baltmore, MD, July 2003. 31