Video inpaining of complex scenes based on local saisical model Voronin V.V. (a), Sizyakin R.A. (a), Marchuk V.I. (a), Yigang Cen (b), Galusov G.G. (c), Egiazarian K.O. (d) ; (a) Don Sae Technical universiy, Dep. of Radio-Elecronics Sysems, Gagarina 1, Rosov on Don, Russian Federaion; (b) Insiue of Informaion Science, Beijing Jiaoong Universiy, Beijing, China; (c) Souhern Federal Universiy, Nekrasovski 44, Taganrog, Russian Federaion; (d) Dep. of Signal Processing, Tampere Universiy of Technology, Korkeakoulunkau 1, Tampere, Finland FI-33720 Absrac This paper describes a framework for emporally consisen video compleion. Proposed mehod allow o remove dynamic objecs or resore missing or ained regions presen in a video sequence by uilizing spaial and emporal informaion from neighboring scenes. The algorihm ieraively performs following operaions: achieve frame; updae he scene model; updae posiions of moving objecs; finding a se of descripors ha encapsulae he informaion necessary o reconsruc a frame; replace pars of he frame occupied by he objecs marked for remove wih use of a 3D paches. In his paper, we exend an image inpaining algorihm based exure and srucure reconsrucion by incorporaing an improved sraegy for video. Our algorihm is able o deal wih a variey of challenging siuaions which naurally arise in video inpaining, such as he correc reconsrucion of dynamic exures, muliple moving objecs and moving background. Experimenal comparisons o sae-of-he-ar video compleion mehods demonsrae he effeciveness of he proposed approach. Inroducion The problem of auomaic video reconsrucion in general, and auomaic objec removal and modificaion in paricular, is beginning o arac he aenion of many researchers. This is problem refers o a field of compuer vision ha aims o remove objecs or resore missing or ained regions presen in a video sequence by uilizing spaial and emporal informaion from neighboring scenes. Video signals ofen conain saic images which may hide some useful informaion. There are a lo of examples of such images like differen channel logos, dae, ime or subiles ha are superimposed on he video wih furher coding. A possible applicaion of video inpaining echniques may be he concealmen of errors and los blocks in decoded bi sreams caused by lossy compression performed by a video coder and media daa ransmission arifacs. In some cases here may be sophisicaed video removal (of undesired saic or dynamic objecs) by compleing he appropriae saic or dynamic background informaion on he video sequence. Here, he erm objec refers o a conneced region of pixels. The example of such objec can be a moving car or person, he defec caused by a scrach on he film or he enire background scene. The ask of video repairing is relaed o he problem of image inpaining. The generic goal of replacing areas of arbirary shapes and sizes in images by some oher conen was firs presened by Masnou and Morel in [1]. This mehod used level-lines o disocclude he region o inpain. The erm inpaining was firs inroduced by Beralmio e al. in [2]. A number of algorihms for auomaic sill image compleion have been proposed in he lieraure [2-5]. There has also been some preliminary work on frame-by-frame parial differenial equaions (PDEs) based video inpaining [3]. This mehod does no ake ino accoun he emporal informaion ha a video provides, and is applicaion is hereby limied. This approach is jusified in removing he defecs of he film. Many ypes of defecs appear only on one frame, and absen from his neighbors. The virue of his mehod is is simpliciy. This mehod is applicable only o small objecs, is applicaion o large areas leads o unsaisfacory resuls. Anoher group of an image inpaining mehods is relaed field of exure synhesis. For exure synhesis he region can be much larger wih he main focus being he filling in of wo-dimensional repeaing paerns ha have some associaed sochasiciy i.e. exures. Srucural properies, such as edges of an objecs, are exraced from he spaial domain and used o complee an objec wih is srucural propery exended [5]. Some of an image inpaining echniques can complee holes based on boh spaial and frequency feaures [6]. Subsequenly, a vas amoun of research was done in he area of image inpaining, and o a lesser exen in video inpaining. The only difference is he necessiy o mainain emporal coninuiy in addiion o he spaial coninuiy. The emporal informaion is eiher considered by using a segmenaion of video objecs (e.g. racking) or a global coherency in space-ime paches. Exising mehods of video inpaining can be divided ino several classes: 1) There are approaches similar o mehods of saic images inpaining. The main varieies: he mehods based on soluion of parial differenial equaions in parial derivaives (PDE); he mehods based on orhogonal ransformaions; he mehods based on exure synhesis. 2) Mehods ha use he space-ime recovery. Provide good qualiy resoraion, bu usually quie cosly in erms of compuaion. 3) Mehods, separaing he original video sequence o a se of layers (in simple case background and foreground), each layer of resoring individually and perform compound-reaed layers. The fac ha video inpaining dealing wih moving objecs in ime and mus consider no only he spaial coninuiy of such objecs, bu also heir emporal coninuiy. In his regard, a simple applicaion of inpaining approaches designed for images sequenially o each frame leads o unsaisfacory resuls. One of problems is he appearance of so-called "ghoss" [7]. A small change of lighing or he movemen of surrounding pixels can lead o a significan change in he resul of recovery. The problem of video inpaining can be divided ino he following caegories [7]: saionary video wih moving objecs; non-saionary video wih sill objecs; non-saionary video wih moving objecs (could be occluded), including all camera moions. IPAS-193.1
A mehod for space-ime compleion of damaged areas in a video as a global opimizaion problem was proposed in [8]. This work exends he echnique of nonparameric sampling [9]. The filling of he hole can be performed based on globally opimized mehod also. The approach proposed by Wexler e al. [10] solves he video inpaining problem of saic camera videos based on he opimizaion of a well-defined energy funcion. The inpaining is performed by opimizing he cos funcion expressing he local consisencies by using a weigh of he global compleion qualiy provided by each possible pixel value. The main disadvanage of his approach is he assumpion ha objecs move in a periodic manner and also hey do no significanly change scale. Also, he camera is saic for his mehod and he processing ime is high even for low resoluions videos. A probabilisic video inpaining mehod has been proposed in [11]. In his mehod define epiomes as pach based probabiliy models ha used o synhesize daa in he areas of video damage or objec removal. The approach is very compuaional complexiy and more suiable for low-resoluion videos. A mehod for repairing damaged video has been proposed in [12] bu i is semi-auomaic approach. The user has o manually draw he boundaries of he differen deph layers of he sequence. For reconsrucion moving objecs use synhesis process and calculaion inerpolaed rajecory. A relaed algorihm, also combining moion layer esimaion and segmenaion, has been repored in [13]. The complexiy of he search for he bes maching paches has been reduced in [14] by using an exension of he pach maching algorihm o he video. The mehod proposed in [15] is a global opimizing inpaining approach wih low compuaional complexiy by racking every pixel, bu his approach can handle only ranslaional or periodic objecs moions. The approach proposed in [16] searches he opimal displacemens so-called shif map which is a vecor fields of he correspondences beween missing pixels and heir corresponding unoccluded values. The mehod inroduced in [17] reconsrucs he moion of people in videos based on he paches similariy in erms of exure and moion. This approach allows reducing he ime complexiy of he paches maching search based on 2D skeleon model of each racked person in he video. I is shows correc resuls only for cyclic objec moions. Oher approaches based on ransfer of moion fields ino he missing area by propagaing moion vecors [18] or by using moion similariies beween paches [19]. These mehods are likely o suffer from smoohing arifacs afer few frames making he approach no well suied for compleion over a high number of frames. The approach based on video segmenaion ino moving foreground objecs and background has been inroduced by Pawardhan e al. in [20] by exending examplar-based image inpaining approach. To inpain he saionary background a relaively simple spaio-emporal prioriy scheme is employed where undamaged pixels are copied from frames emporally close o he damaged frame, followed by a spaial filling in sep which replaces he damaged region wih a bes maching pach so as o mainain a consisen background hroughou he sequence. This algorihm provides high-qualiy visual recovery, bu demanding of compuing resources in he search for similar paches. This approach was exended o process video sequences in [21] where he auhors have aemped o provide boh spaial and emporal coninuiy. Searching of similar pach is performed no only on he curren frame, bu hroughou he video sequence, or in some bounded area of i. In [22-24] have been made some ries o use various opimizaions: objec racking, mosaic images, separaion of video sequence o se of moving. The main drawbacks of he known mehods come from he fac ha he mos of hem are unable o recover he curved edges of objec. I should be also noed ha hese mehods ofen blur image in he recovery of large areas wih missing pixels. Addiionally, he mehods suffer from he lack of global visual coherence especially for large holes. For mos of he mehods, boh periodic moion for each occluded objec and accurae segmenaion of moving objecs and saic background are ofen necessary o provide pleasing resuls. Oherwise, segmenaion errors lead o severe arifacs. Furhermore, if he inpaining of moving objecs is performed independenly of he background, he blending beween he compleed foreground and he background can look unnaural. Mos of hese mehods are compuaionally very demanding and inappropriae for implemenaion on modern mobile plaforms. In his paper we propose a framework for video reconsrucion, aimed a achieving high-qualiy resuls in he conex of film posproducion. Our proposed mehod builds on exising exemplarbased echniques and exends hem o process videos. We propose o use a se of descripors ha encapsulae he informaion of periodical moion of objecs necessary o reconsruc missing/corruped frames. For background resoraions used se of 3D paches. The proposed video inpaining mehod A discree frame defined on a I J recangular grid is denoe i 1, I, j 1, J, T Yi, j, and can be represened as follows: Y i, j ( 1 Mi, j ) Si, j Mi, j Ri, j, where S i, j are he rue image pixels; M M i, j is a binary mask of he disored values of pixels (1 - corresponds o he missing pixels, 0 - corresponds o he rue pixels); R i, j are missing pixels; I is he number of rows, J is he number of columns and T is he number of he frames. In his aricle we will discuss he video inpaining proposal pu forward by Pawardhan e al. [21]. The special feaure of his mehod is he abiliy o resore he video, sho by a moving camera. In fac, his mehod is a generalizaion of he exemplar based mehod in case of video sequences ha adds o he spaial coninuiy he ime. Recovery area may be differen: moving objec, saic objec and oher. I can be background or foreground objec. I can be blocked by oher objecs or can block hem. The algorihm includes preprocessing sage and he wo work phases. A he preprocessing sage is performed a rough segmenaion of each frame in he foreground and he background. Afer his sep some regions can sill be empy - for is resoraion is used o search for similar blocks of he curren frame. This algorihm has some disadvanages. Searching paches in he exure resoraion requires significan compuaional complexiy o resore large exure areas. The exemplar-based mehods use a non-parameric sampling model and exure synhesis. We will ackle his problem by firs sage resoraion using a se of descripors ha encapsulae he informaion of periodical moion of objecs. The diagram of he proposed mehod is shown a Figure 1. IPAS-193.2
Pre-processing Saic Background Moion Inpaining Clipping Compue he highes prioriy locaion for moion filling- in Search in he global video descripor o find he possible candidae frames - Search each candidae clip frames for bes 3D pach - Copy moving objec from mach o damaged frame, and - Consrain prioriies for nex ineraion... Global video descripor calculaion he scene model; updae posiions of moving objecs; finding a se of descripors ha encapsulae he informaion necessary o reconsruc a frame; replace pars of he frame occupied by he objecs marked for remove wih use of a 3D paches. A he preprocessing sage is performed a rough segmenaion of each frame in he foreground and he background. The main idea of he proposed approach is o, raher han direcly aemp o inerpolae missing pixels, esimae, based on all available spaio emporal informaion, he value of a se of descripors ha encapsulae he informaion necessary o reconsruc missing/corruped frames. In order o esimae he values of descripors, we will collec he values of all he descripors corresponding o he Fh clip in a vecor fk. For his purpose we use he global descripor is compued by applying a bank of 3D spaioemporal filers on he frequency specrum of a video sequence for inegraes he informaion abou he moion and scene srucure. In his aricle we use a descripor pu forward by Solmaz e al. [25] which describes an approach o classify realisic videos of differen acions. The descripor is generaed by applying a bank of 3D spaioemporal filers on he frequency specrum of a sequence (Fig. 2). The bandpass naure of hese filers alleviaes he need for moion compensaion. Furhermore, as opposed o he approaches which apply bag-of-feaures model, he approach preserves he spaial and emporal informaion, as we perform quanizaion in fixed spaio-emporal sub-volumes afer applicaion of each filer on he frequency specrum and aking he inverse Fourier ransform. As he filer responses for all filers on all sub-volumes are concaenaed, he ordering and he lengh of each feaure vecor are idenical for all video clips. The framework of his approach is shown in Fig. 3. Coninue unil all pixels eher are filled -in or have zero prioriy Repea his block for all frames requiring moion inpaining Background Filling- in Temporal copying of 3D pach or Prioriy based exure synhesis o fill in he remaining hole Figure 2. A bank of 3D spaio-emporal filers Figure 1. Algorihm of video inpaining mehod Proposed approach allow o remove objecs or resore missing or ained regions presen in a video sequence by uilizing spaial and emporal informaion from neighboring scenes. The algorihm ieraively performs following operaions: achieve frame; updae The frequency specrum compued for a video clip could capure boh scene and moion informaion effecively, as i represens he signal as a sum of many individual frequency componens. In a video clip, he frequency specrum can be esimaed by compuing he 3-D discree Fourier ransform (DFT). The moion is an imporan elemen which can be represenaive of he ype of performed acion in a scene. The frequency specrum of a wodimensional paern ranslaing on an image plane lies on a plane, he orienaion of which depends on he velociy of he paern. IPAS-193.3
3D DFT 3D filer bank 3D IDFT Averaging Figure 5. Pseudocode for he moion inpaining sep PCA Figure 3. Algorihm of global video descripor calculaion The global video descripor allows finding frames wih similar movemen. The foreground objecs o be inpained exhibis repeiive moion 3D paches (Fig. 4). The parial objecs are firs compleed wih he appropriae objec emplaes seleced by minimizing a window-based dissimilariy measure. Beween a window of parially-occluded objecs and a window of objec emplaes from he daabase, we define he dissimilariy measure as he Sum of he Squared Differences (SSD) in heir overlapping region. The firs sep in reamen is he resoraion of moving foreground objecs, which "overlap" he resored area. Afer ha here is recovery of he remaining area by copying blocks from adjacen frames. Afer his sep some regions can sill be empy - for is resoraion is used o search for similar blocks of he curren frame. This 3D pach searching is implemened using he following seps (refer o Fig. 5). The proposed mehod has he following advanages over currenly exising echniques: i leads o a non ieraive, compuaionally aracive algorihm ha opimizes he use of (global) spaio/emporal and dynamics informaion and has a moderae compuaional burden; i is no resriced o he case of periodic moion, saic background or saionary cameras; i can be used o exrapolae frames, ha is exend a given video sequence, and, in he case of dynamic exures. Experimenal resuls The effeciveness of he presened scheme is verified on he es frames of a video sequence wih missing pixels presened. Afer applying he missing mask, all frames have been inpained by he proposed mehod and sae-of-he-ar mehods [8, 14]. In his example we will consider he problem of inpaining dynamic exures, e.g. sequences whose frames are relaively unsrucured, bu possessing some overall saionary properies. In Figures 6-9 examples of video compleion (a - he image wih a missing pixels, b - he resoraion by he Wexler mehod, c - he resoraion by he Newson mehod, d - he resoraion by he proposed mehod) are shown. I should be observed ha our echnique compares favorably even in he presence of he moderae dynamic background. Figure 4. 3D paches searching Figure 6. Examples of video inpaining IPAS-193.4
Conclusion The paper presens an video inpaining algorihm of complex scenes based on he exure and srucure reconsrucion of video sequence. The background is filled-in by exending spaial exure synhesis echniques based on local saisical model. Examples presened in his paper demonsrae he effeciveness of he algorihm in resoraion of saic background and moving foreground of he video sequences having differen geomerical characerisics. Acknowledgmen The repored sudy was suppored by he Russian Foundaion for Basic research (RFBR), research projecs 15-07-99685, 15-01-09092, 15-37-21124. Figure 7. Examples of video inpaining References [1] S. Masnou and J.-M. Morel, Level lines based disocclusion, in In. Conf. Image Processing (ICIP), 1998. [2] M. Beralmio, G. Sapiro, V. Caselles, and C. Balleser, Image inpaining, in Proc. ACM SIGGRAPH, pp. 417 424, 2000. [3] M. Beralmio, A. L. Berozzi, and G. Sapiro, Navier-sokes, fluid dynamics, and image and video inpaining, in Proc. IEEE Compuer Vision Paern Recogniion, vol. 1, pp. 355 362, 2001. [4] C. Balleser, V. Caselles, and J. Verdera, Dissoclusion by join inerpolaion of vecor fields and gray levels, SIAM Muliscale Model. Simul., vol. 2, pp. 80 123, 2003. [5] A. Criminisi, P. Perez, and K. Toyama, Region filling and objec removal by exemplar-based inpaining, IEEE Trans. Image Process., vol. 9, no. 9, pp. 1200 1212, 2004. Figure 8. Examples of video inpaining [6] T.F. Chan, J. Shen, "Mahemaical models of local non-exure inpainings," SIAM J. Appl. Mah, vol. 62(3), pp. 1019-1043, 2002. [7] Timohy K. Shih, Nick C. Tang, and Jenq-Neng Hwang, Exemplar-Based Video Inpaining Wihou Ghos Shadow Arifacs by Mainaining Temporal Coninuiy, IEEE ransacions on circuis and sysems for video echnology, vol. 19, no. 3, pp. 347-360, 2009. [8] Y. Wexler, E. Shechman, and M. Irani, Space-ime video compleion, in Proc. IEEE Compu. Soc. Conf. Compuer Vision and Paern Recogniion, vol. 1, pp. 120 127, 2004. [9] A.A. Efros and T.K. Leung, Texure synhesis by nonparameric sampling, presened a he IEEE In. Conf. Compuer Vision, Corfu, Greece, 1999. [10] Y. Wexler, E. Shechman, and M. Irani, Space-ime compleion of video, IEEE Trans. on Paern Analysis and Machine Inelligence (PAMI), pp. 463 476, 2007. Figure 9. Examples of video inpaining [11] V. Cheung, B. J. Frey, and N. Jojic, Video epiomes, in IEEE Conf. Compuer Vision and Paern Recogniion, vol. 1, pp. 42 49, 2005. [12] J. Jia, T. Wu, Y. Tai, and C. Tang, Video repairing under variable illuminaion using cyclic moions, in Proc. IEEE Compuer Soc. Conf. Compuer Vision and Paern Recogniion, vol. 1, pp. 364 371, 2004. IPAS-193.5
[13] Y. Zhang, J. Xiao, and M. Shah, Moion layer based objec removal in videos, in Proc. Workshop on Applicaions of Compuer Vision, pp. 516 521, 2005. [14] A. Newson, A. Almansa, M. Frade, Y. Gousseau, and P. Perez, Towards fas, generic video inpaining, in Proceedings of he 10h European Conference on Visual Media Producion, pp. 7:1 7:8, 2013. [15] Y. Shen, F. Lu, X. Cao, and H. Foroosh, Video compleion for perspecive camera under consrained moion, in IEEE In. Conf. on Image Proc. (ICIP), vol. 3, pp. 63 66, 2006. [16] Y. Hu and D. Rajan, Hybrid shif map for video reargeing, in In. Conf. on Compuer Vision and Paern Recogniion (CVPR), pp. 577 584, 2010. [17] T. Shih, N. Tan, J. Tsai, and H.-Y. Zhong, Video falsifying by moion inerpolaion and inpaining, in In. Conf. on Compuer Vision and Paern Recogniion (CVPR), pp. 1 8, 2008. [18] Y. Masushia, E. Ofek, W. Ge, X. Tang, and H.-Y. Shum, Full-frame video sabilizaion wih moion inpaining, IEEE Trans. on Paern Analysis and Machine Inelligence (PAMI), pp. 1150 1163, 2006. [19] T. Shiraori, Y. Masushia, X. Tang, and S. Kang, Video compleion by moion field ransfer, in In. Conf. on Compuer Vision and Paernn Recogniion (CVPR), pp. 411 418, 2006. [20] K.A. Pawardhan, G. Sapiro, and M. Beralmio, Video inpaining of occluding and occluded objecs, in Proc. IEEE In. Conf. Image Process., vol. 2, pp. 69 72, 2005. [21] K.A. Pawardhan, G. Sapiro, and M. Beralmío, Video inpaining under consrained camera moion, IEEE Trans. Image Process., vol. 16 (2), pp. 545 553, 2007. [22] J.-F. Aujol, S. Ladjal and S. Masnou, "Exemplar-based inpaining from a variaional poin of view, SIAM J. Mah. Anal., vol. 44, pp. 1246 1285, 2010. [23] F. Cao, Y. Gousseau, S. Masnou, P. Pérez, Geomerically Guided Exemplar-Based Inpaining, SIAM J. Imaging Sci., vol. 4(4), pp. 1143 1179, 2011. [24] V.V. Voronin, V.I. Marchuk, N.V. Gapon, A.V. Zhuravlev, S. Maslennikov, S. Sradanchenko, Inpaining for videos wih dynamic objecs using exure and srucure reconsrucion, Proc. SPIE 9497, Mobile Mulimedia/Image Processing, Securiy, and Applicaions, 94970Y, 2015. [25] B. Solmaz, S.M. Assari, M. Shah, Classifying web videos using a global video descripor, Machine Vision and Applicaions, vol. 24 (7), pp. 1473-1485, 2013. Auhor Biography Viacheslav Voronin was born in Rosov (Russian Federaion) in 1985. He received his BS in radio engineering from he Souh-Russian Sae Universiy of Economics and Service (2006), his MS in radio engineering from he Souh-Russian Sae Universiy of Economics and Service (2008) and his PhD in echnics from Souhern Federal Universiy (2009). Voronin V. is member of Program Commiee of conference SPIE. His research ineress include image processing, inpaining and compuer vision. Sizyakin Roman was born in Rosov in 1989. He recieved his BS in elecrical engineering from Souh-Russian universiy of economics and service in 2011. He recieved he MS degree in echnics from Don Sae Technical Universiy (Russian Federaion) in 2013. A he ime he is a PhD suden a Don Sae Technical Universiy. His research ineress lay in he areas of digial image processing and compuer vision. Vladimir Marchuk was born in 1951. He received he D.Tech. degree in echnics from Souhern Federal Universiy (Russian Federaion) in 2006. Since 2006, he has been a Professor. His research ineress are in he areas of applied saisical mahemaics, signal and image processing. Yigang Cen was born in 1978. He received he Ph.D. degree a School of Compuer & Informaion Technology, Beijing Jiaoong Universiy. His research ineress are in he areas of applied mahemaics, signal processing, and digial logic. Gennady Galusov was born in 1941. He defended PhD hesis in cyberneics in 1971, and a docorae hesis in compuer-aided simulaion of biological signals in 1991 (boh in TRTI). He was a researcher and lecurer in he heory of random processes TRTI. His research ineress are in he field of developmen of he heory of elecrical signals, he synhesis of random processes, research and reamen of biomedical signals. Karen Egiazarian (SM 96) was born in Yerevan, Armenia, in 1959. He received he M.Sc. degree in mahemaics from Yerevan Sae Universiy in 1981, he Ph.D. degree in physics and mahemaics from Moscow Sae Universiy, Moscow, Russia, in 1986, and he D.Tech. degree from he Tampere Universiy of Technology (TUT), Tampere, Finland, in 1994. He has been Senior Researcher wih he Deparmen of Digial Signal Processing, Insiue of Informaion Problems and Auomaion, Naional Academy of Sciences of Armenia. Since 1996, he has been an Assisan Professor wih he Insiue of Signal Processing, TUT, where he is currenly a Professor, leading he Specral and Algebraic Mehods in DSP group. His research ineress are in he areas of applied mahemaics, signal processing, and digial logic. IPAS-193.6