FAST MOTION BLUR COMPENSATION IN HEVC USING FIXED-LENGTH FILTER. Yiqun Liu and Jörn Ostermann

Transcription

1 FAST MOTION BLUR COMPENSATION IN HEVC USING FIXED-LENGTH FILTER Yiqun Liu an Jörn Ostrmann Institut für Informationsvrarbitung, Libniz Univrsität Hannovr Applstr. 9A, 3067 Hannovr, Grmany ABSTRACT Motion compnsation is on of th most important lmnts in morn hybri vio cors. It utilizs tmporal information to prict th currnt block an rucs thrby th runancy of a vio. Th priction accuracy pns on th similarity btwn th rfrnc block an th currnt block. It is cras by varying motion blur caus by th acclration of th camra or crtain objcts in a scn. Thus, w mploy fix-lngth filtrs to compnsat varying motion blur in hybri vio coing. Whil formr approachs n aitional signaling for blurring filtrs or a scon motion stimation, our algorithm rivs th blurring filtr only bas on th motion vctor an ns only on motion stimation. W implmnt our approach in th High Efficincy Vio Coing (HEVC) rfrnc softwar HM-3.0. Compar to th rfrnc HM-3.0, w gain 2.54% in trms of BD-Rat in avrag for JCT-VC tst squncs an 4.5% for slf-rcor squnc containing lots of varying motion blur, with limit incras in coing tim. Inx Trms Motion Blur, Vio Coing, HEVC. INTRODUCTION f g b c a c θ Motion compnsation togthr with intra-fram priction, quantization, transform an ntropy coing has bn th cornrston of hybri vio coing systm. It is us in th High Efficincy Vio Coing (HEVC) [] stanar as wll as in prcssors lik MPEG-2 [2] an AVC [3]. Motion compnsation uss alray co prvious or futur frams to prict th contnt of a block, call coing unit (CU) in HEVC. Insta of th original CU contnt, only a isplacmnt vctor call Motion Vctor () an th corrsponing priction rror ar us for coing an quantization, in orr to gnrat th bit stram. Th ata rat can b vastly ruc by this priction tchniqu if th similarity btwn th rfrnc fram an th currnt fram is high. Th common motion compnsation (MC) mtho from vio coing stanars works wll with stationary objcts or moving objcts of constant vlocity. Howvr, th accuracy is limit in cas of varying motion blur. Motion blur occurs in th irction of th objct motion if an objct movs uring th xposur tim. Th changing xtnt of blur btwn sucb g f Cofficints a = 8/56 b = /56 c = 7/56 = 3/56 = 5/56 f = 5/56 g = 3/56 Fig. : Discrt vrsion of linar motion blur (θ = arctan 4 5 ) cssiv frams gnrally rucs th comprssion fficincy bcaus of th incras priction rror. Svral approachs wr propos to ruc th priction rror by filtring th rfrnc fram for varying motion blur compnsation. Som of thm us ithr pr-fin filtr [4][5] or aaptiv filtr [6] for singl layr coing. All thr approachs n aitional signaling for th choic of filtr or for th filtr cofficints, rspctivly. Othr proposals avoi filtr signaling by ithr xploiting information provi by th bas layr in th contxt of scalabl vio coing [7] or introucing a sctor structur [8]. Howvr, in both cass th numbr of blurring filtrs is limit or prfin an two motion stimations (ME) will b xcut in th procss. Consquntly th accuracy of th filtr is rstrict an long coing tim is n. In this papr w propos a blur compnsation algorithm for singl layr coing that gnrats th blurring filtr in arbitrary irction bas only on th irction of th transmitt. W apply th intical for MC bas on th filtr rfrnc an spar on ME. No aitional signaling for th filtr cofficints is ncssary, whras th usag of th filtr is signal by th ncor for ach CU in th bit stram. Th rmaining papr is organiz as follows: Sction 2 analyzs th filtr us for motion blur compnsation an xplains our mtho in tail. Exprimntal rsults ar givn in Sction 3 bfor Sction 4 conclus th papr.

2 Tabl : Discrt form of propos 3 3 blurring filtr for iffrnt θ [0, π) θ [ [ 0, arcsin 3) arcsin 3, arccos ) [ 3 arccos 3, π arccos ) [ 3 π arccos a 0 3, π arcsin 3 ) [ π arcsin 3, π) 2 6 min(sin θ, cos θ) 0 ( b 0 max(0, 6 cos θ ) sin θ ) 2 ( 6 sin θ max(0, 6 cos θ + ) sin θ ) 0 c min(sin θ,cos θ) 0 2 ( 6 cos θ max(0, 6 sin θ ) ( cos θ ) 0 max(0, 6 sin θ + ) cos θ ) cos θ 3 max( cos θ,sin θ) Tabl 2: Filtr cofficints Enco CU 3 3 Filtr θ = 5 θ = 60 a b c c b a Mrg/Skip Motion Estimation 2. PROPOSED REFERENCE FRAME FILTERING Blurring Flag Tru Du to th variability of th xtnt of motion blur which is caus by th chang of th vlocity of th camra or of an objct btwn frams, w suggst to filtr th rfrnc fram in orr to a blur an incras th similarity btwn th rfrnc CU an th currnt CU. A commonly us linar uniform motion blur point spra function (PSF) in continuous tim omain is scrib with a lin sgmnt L an an angl θ with rspct to th horizontal axis [9], as givn in Eq. (): INTRA SKIP Blurring Flag Fals Motion Compnsation INTER Blur Compnsation Filtr Gnration Blurring Rfrnc Fram h(x, y) = { L, x2 + y 2 L 2, y x = tan θ 0, othrwis () Rat-Distortion Optimization L is proportional to th motion sp an th xposur uration, θ inicats th motion irction an (x,y) is th location in th Cartsian coorinat systm. Th iscrt vrsion of Eq. () is acquir by consiring a bright spot travrsing across th snsors covr by th lin sgmnt uring th xposur tim with constant vlocity [0]. Each cofficint of th blur krnl is proportional to th tim spnt on ach snsor lmnt. With th assumption of a constant motion, th filtr cofficints ar givn by th normaliz lngth of th intrsction of th lin sgmnt with ach pixl in th gri, as illustrat by Fig.. In cas of varying motion blur, a filtr rfrnc may improv th coing prformanc. W assum that motion blur as wll as th chang of motion blur can b scrib by Eq. (). Motion can b consir constant for th xposur tim of a singl fram. Sinc th tim intrval btwn two narby frams is vry short,.g scons for a 50 fps squnc, w suggst that th chang of motion blur xtnt is small in most cass. Hnc, w mploy a fix xtnt of 3 pls to scrib th phnomnon of variation in blurring, i.., L = 3. Fig. 2: Flow chart of motion blur compnsation Th two imnsional blurring filtr has a imnsion of 3 3. W blur only th luminanc componnt of th rfrnc, sinc chrominanc pixls li at vry othr pixl position of th luminanc componnt [] for a 4:2:0 squnc. Th othr gr of from for a blurring filtr is th angl θ. It is riv from th foun in non-blur cas. Hnc a stanar ME in HEVC is xcut bfor th blurring filtr is stablish. Th blurr rfrnc fram is gnrat by using th irctions of th s. Th gnral iscrt vrsion of our 3 3 filtr has 5 iffrnt cofficints (a ) u to symmtry an is calculat accoring to an arbitrary angl θ [0, π) using Tabl. Th suggst filtr is a low-pass filtr. As an xampl, filtr cofficints for θ = 5 an θ = 60 ar list in Tabl 2. Bsis th stanar coing mthos lik Intra-fram an Intr-fram priction (incl. Skip), w a a motion blur compnsation mo to HEVC. This motion blur compnsation

3 (a) Dco Fram (b) Pricion Mo Fig. 3: Coing mo istribution of Basktball Driv. R (arkst): Blur, Grn: Skip, Yllow: Intr an Whit: Intra. can b consir as a complmnt of Intr mo (incl. Skip) of HEVC uring th coing of a crtain CU, as illustrat in Fig. 2. W crat tmporal rfrnc frams by filtring th rconstruct fram with a filtr of angl θ riv from th an us ths rfrnc frams for motion blur compnsation. Th tmporal rfrnc frams ar paralll to th rconstruct ons, i.., on rfrnc inx rfrs to ithr th rconstruct fram for gnral Intr-fram priction or th blurr rfrnc fram for motion blur compnsation. For ach fram in th rfrnc pictur list thr is a corrsponing fram for motion blur compnsation, sinc th rsults of ME bas on rconstruct frams an blurr frams o not always shar th sam rfrnc inx. Compar to [8], w simplify th ncoing structur by applying th sam to MC bas on filtr rfrnc insta of a scon ME. Manwhil it is highly likly that a CU will b bttr prict using our propos mtho. Th rasons ar list blow.. Objct shoul mov in th irction whr variation of blur xists. W crat th tmporal rfrnc xactly in th irction givn by without any moification. 2. Our Blurring filtrs ar always symmtrical, which mans no phas shift by filtring. It is probabl that th majority of th s will not chang with a scon ME bas on thos blurr tmporal rfrncs. A si bnfit is th ruction of coing tim. W spar on aitional ME an hav smallr rgion of blurring. In plac of blurring th whol ara covr by stanar ME, w blur only th CU on rfrnc point by th givn an its surrouning which will b us by intrpolation latr. A Blurring Flag will b st within th ncoing for ach CU which is co with th Intr mo (incl. Skip) for all partition sizs. Th cor is abl to trmin whthr th rfrnc fram is blurr uring rconstruction of a CU an to rplicat th blurring of th rfrnc fram using th angl θ riv from th transmitt. Th blurring flag is consir within th Rat-Distortion optimization (RDO). Tabl 3: Distribution of on filtr rfrnc Squnc Intical ± Pl ±2 Pl Playgroun 70.72% 97.29% 98.40% Bik % 9.63% 92.48% Bik % 9.63% 93.00% Bik % 86.3% 87.86% Rcor Avg % 9.56% 92.93% Basktball Driv 77.59% 88.3% 89.58% Kimono 75.64% 9.04% 92.37% Cactus 74.96% 93.33% 95.25% Park Scn 69.4% 92.64% 94.38% BQ Trrac 62.49% 97.90% 98.84% Popl On Strt 59.52% 90.6% 92.3% Traffic 64.% 96.48% 98.50% JCT-VC Avg % 92.90% 94.44% Avrag 70.87% 92.4% 93.89% 3. EXPERIMENTAL RESULTS Th implmntation of our propos algorithm is bas on th HEVC rfrnc softwar HM-3.0 [2]. W hav appli th JCT-VC common tst conitions [3] with th fault configuration of Low Dlay P (LD-P) an Ranom Accss (RA) from HM-3.0 for th valuation. Our tst st inclus JCT-VC squncs (Basktball Driv, BQ Trrac, Cactus, Kimono, Park Scn, Popl On Strt, Traffic) as wll as slf-rcor squncs (280 x 720). Th lattr inclu Playgroun [4] (film with a han-hl camra) an Bik (film with a camra attach to th hlmt of a cyclist) [7]. W us a fix focal lngth for all slf-rcor squncs. Thus no blur contain in th squncs is caus by focus chang. Th usag of motion blur compnsation is illustrat in Fig. 3 which illustrats fram 437 of Basktball Driv as an xampl. In Fig. 3b, CUs co with th propos blurring mtho ar shown in r whil yllow, grn an whit rprsnts intr, skip an intra mos, rspctivly. Looking at th

4 co fram (Fig. 3a), motion blur compnsation is chosn mostly for th aras whr th obvious varying motion an rlat blur shoul occur: on th bois of th playrs. Tabl 3 shows th istribution of th nw bas on th blurr rfrnc frams if w us a stanar ME aftr blurring. Intical inicats that th s bas on th tmporal rfrnc ar xactly th sam as th s from th rconstruct non-blurr rfrnc. Column ± an ±2 show th ratio of nw s, which hav lss than an 2 full pls viation to th original in ach irction, rspctivly. For both JCT-VC an slf-rcor squncs aroun 70% of th s ar intical, which mans that only about 30% CUs hav lss goo priction than a stanar sarch. A furthr 23.84% an 7.52% CUs hav th bst prictor insi -pl rang of givn. Th positional viation is subjctivly unnoticabl whil objctivly it has small impact on th priction unlss som ramatic changs occur in th scn. Hnc, th priction inaccuracy brought by irctly using th as motion sarch rsult insta of a scon ME is small an it coul b compnsat by th improvmnt of filtr accuracy. Th simulation rsult is prsnt in trms of Bjøntgaar lta (BD)-Rat [5]. Ngativ numbrs man gain compar to th anchor whil positiv numbrs rprsnt a loss. As an anchor w us HM-3.0 without motion blur compnsation. Our slf-rcor squncs contain much mor varying motion blur u to th camra motion as wll as longr xposur tim. Hnc thy bnfit mor from our propos motion compnsation mtho, as shown in Tabl 4. Th avrag gain of ths squncs compar to th HM-3.0 for {Y, U, V } rachs {4.5%, 2.44%, 2.26%} for LD-P configuration, majority of which offr ovr 4% gain for luminanc. As xpct, th propos mtho works lss goo on JCT-VC squncs. Nvrthlss, w gt gain also for all thos squncs, which is avrag to {2.54%, 0.50%, 0.0%} for {Y, U, V } rspctivly. Bcaus of th rapi chang in motion blur from th playrs of Basktball Driv an from th lavs on backgroun of Kimono, it is rasonabl to obsrv highr luminanc gains from thm in this catgory. Th last column of Tabl 4 shows th ratio of CUs that choss blurring mo aftr RDO with LD-P configuration. Th gain basically incrass with th ris of th ratio in ach st of squncs. Th valus for Basktball Driv an Kimono prov that motion blur compnsation works wll for ths two in th st. Slf-rcor squncs work bttr with narly th sam lvl of ratio, sinc thy hav mor shaking an nois, which mak ths squncs mor snsitiv to th low pass charactristic of th blurring filtr. Ovrall, th BD-Rat gain of motion blur compnsation is {3.26%,.20%, 0.89%} on avrag for LD-P with 7% CUs staning by this mtho an {0.95%, 0.38%, 0.33%} for RA. Rsults bas on LD-P ar bttr than that bas on RA, sinc th tmporal istanc btwn th rfrnc fram an th currnt fram from LD-P is shortr an th propos filtr is Tabl 4: BD-Rat vs. HM-3.0 an Ratio of blur in CUs Squnc Low Dlay P Y U V Blur Playgroun -4.28% -2.9% -4.2% 8.25% Bik % -2.68% -.54% 8.30% Bik % -.05% -0.64%.00% Bik % -3.0% -2.72% 26.67% Slf-Rcor -4.5% -2.44% -2.26% 8.56% Basktball Driv -4.24% -.39% -.04% 24.75% Kimono -4.06% 0.93% 0.78% 35.64% Cactus -2.7% -0.76% -0.7% 2.00% Park Scn -.32% 0.2% -0.76% 7.89% BQ Trrac -2.2% -.08%.8% 4.08% Popl On Strt -2.53% -.02% -0.8% 7.90% Traffic -.35% -0.39% 0.0% 5.30% JCT-VC -2.54% -0.50% -0.0% 6.79% Avrag -3.26% -.20% -0.89% 7.44% Squnc Ranom Accss Y U V Blur Playgroun -2.92% -.09% -.24% 2.64% Bik 0-2.4% -0.9% -0.22% 7.5% Bik % -0.5% -0.36%.4% Bik % -.09% -0.69% 7.88% Basktball Driv -0.88% -0.86% -0.75% 9.22% Kimono -0.27% 0.3% 0.06%.% Cactus -0.30% 0.04% -0.27% 4.0% Park Scn -0.0% 0.07% -0.04% 5.0% BQ Trrac -0.33% -0.03% 0.02% 4.78% Popl On Strt -0.38% -0.65% -0.32% 5.5% Traffic 0.0% 0.23% 0.3% 2.67% Avrag -0.95% -0.38% -0.33% 9.99% only 3 3, which is spcially sign for small variations of motion blur. Coing tim of ncor rops from ovr 500% bas on [8] to 37% of HM-3.0 at th sam tim as a si bnfit, with an aitional 0.28% gain for LD-P configuration. Dcoing tim rops from ovr 200% to about 00% of HM. 4. CONCLUSION Th prformanc of th gnral motion compnsat priction in HEVC is limit by varying motion blur. To improv th priction accuracy in orr to ruc th coing ata rat w propos 3 3 blurring filtrs for rfrnc fram filtring. Th filtr is gnrat bas on purly th arbitrary irction of th transmitt. No xtra signaling of filtr cofficints is n. Compar to HM-3.0, our propos motion blur compnsation can provi an avrag BD-Rat gain of 2.54% for JCT-VC squncs an of 4.5% for consumr rcor squncs, with 37 % ncoing an 00% coing tim.

5 5. REFERENCES [] Bnjamin Bross, Woo-Jin Han, Gary J. Sullivan, Jns- Rainr Ohm, an Thomas Wigan, High Efficincy Vio Coing (HEVC) txt spc. raft 0 (for FDIS&Consnt), in JCT-VC Doc. JCTVC-L003, 2th Mting: Gnva, Switzrlan, Jan [2] ISO/IEC an ITU-T, Rcommnation ITU-T H.262 an ISO/IEC (MPEG-2 Part 2): Information tchnology - Gnric coing of moving picturs an associat auio information: Vio, Mar [3] ISO/IEC an ITU-T, Rcommnation ITU-T H.264 an ISO/IEC (MPEG-4 Part 0): Avanc Vio Coing (AVC)-3r E., Gnva, Switzrlan, July [4] M. Buagavi, Vio Comprssion Using Blur Compnsation, in Imag Procssing, ICIP IEEE Intrnational Confrnc on, Spt 2005, vol. 2, pp. II [2] K. McCann, B. Bross, W. Han, I. Kim, K. Sugimoto, an G. J. Sullivan, High Efficincy Vio Coing (HEVC) Tst Mol 3 (HM 3) Encor Dscription, in JCT-VC Doc. JCTVC-O002, 5th Mting: Gnva, Switzrlan, Oct [3] Xiang Li, Jill Boyc, Patric Onno, an Yan Y, L009: Common Tst Conitions an Softwar Rfrnc Configurations for th Scalabl Tst Mol. Joint Collaborativ Tam on Vio Coing (JCT-VC) of ITU-T SG 6 WP 3 an ISO/IEC JTC /SC 29/WG. 2th Mting, Gnva, CH, 4-23 Jan, 203. [4] Kai Cors, Björn Schurmann, Boo Rosnhahn, an Jörn Ostrmann, Occlusion Hanling for th Intgration of Virtual Objcts into Vio, in Th 7th Intrnational Confrnc on Computr Vision Thory an Applications (VISAPP), Fb. 202, pp [5] Gisl Bjøntgaar, Calculation of Avrag PSNR Diffrncs btwn RD curvs, in ITU-T SG6/Q6 Output Documnt VCEG-M33, Austin, Txas, April 200. [5] Jinik Jang, Hyuk L, Ta-Young Jung, Sung-Min Hong, an Jchang Jong, Pr-filtring with Locally Aaptiv Filtr St for Rfrnc Fram in Vio Comprssion, in Multimia an Expo (ICME), 20 IEEE Intrnational Confrnc on, July 20, pp. 6. [6] Yu. Vatis an J. Ostrmann, Aaptiv Intrpolation Filtr for H.264/AVC, IEEE Transactions on Circuits an Systms for Vio Tchnology, vol. 9, no. 2, pp , [7] T. Lau, H. Mul, Y. Liu, an J. Ostrmann, Motion Blur Compnsation in Scalabl HEVC Hybri Vio Coing, in Pictur Coing Symposium (PCS), 203, Dc 203, pp [8] Y. Liu, W. Wu, an J. Ostrmann, Motion Blur Compnsation in HEVC Using Fix-lngth Aaptiv Filtr, in Pictur Coing Symposium (PCS) 205, submitt for rviw. [9] H. Lin an C. Chang, Photo consistnt motion blur moling for ralistic imag synthsis, in PSIVT 2006, LNCS 439, 2006, pp [0] J.P. Olivira, M.A.T. Figuiro, an J.M. Bioucas- Dias, Paramtric Blur Estimation for Blin Rstoration of Natural Imags: Linar Motion an Out-of-Focus, Imag Procssing, IEEE Transactions on, vol. 23, no., pp , Jan 204. [] M. Ghanbari, Vio Coing, an Introuction to Stanar Cocs., Th Institution of Elctrical Enginrs, Lonon, Unit Kingom, 999.