Auomaic locaion and removal of video logos Wei-Qi Yan 1 Jun Wang 2 Mohan S. Kankanhalli 1 1 School of Compuing Naional Universi of Singapore Singapore e-mail: {anwq mohan}@comp.nus.edu.sg 2 Facul of Elecrical Engineering Mahemaics and Compuer Science Delf Universi of Technolog The Neherlands e-mail: j.wang@ewi.udelf.nl Absrac. Mos commercial elevision channels uilize video logos which can be considered as a form of a visible waermark as a declaraion of inellecual proper ownership. The are also used as a smbol of auhorizaion o rebroadcas when original logos are used in conjuncion wih newer logos. An unforunae side effec of such logos is he concomian decrease in he viewing pleasure. In his paper we uilize he emporal correlaion of he video frames in order o deec and remove he video logos. In he video logo deecion par as an iniial sep he logo boundar bo is firs locaed b uilizing a disance hreshold of video frames and is furher refined b emploing a comparison of edge lenghs. Second our proposed Baesian classifier framework locaes fragmens of logos called logo-les. In his framework we ssemaicall inegrae he prior knowledge abou he locaion of he video logos and heir inrinsic local feaures in order o achieve a robus deecion resul. In our logo removal par afer marking he logo region a maching echnique is emploed o find he bes replacemen pach for he marked region wihin ha video sho. This echnique is found o be useful for small logos. Furhermore we eend he image inpaining echnique o videos. Unlike he use of wo-dimensional gradiens in he image inpaining echnique we inpain he logo region of video frames b using hree-dimensional gradiens eploiing he emporal correlaions in video. The advanage of his algorihm is ha he inpained regions are consisen wih he surrounding eure and hence he resul is percepuall pleasing. We presen he resuls of our implemenaion and demonsrae he uili of our mehod for logos removal. Ke words: Video logo deecion Video logo removal Video inpaining Visual waermark aack - Neural nework 1
1 Inroducion Video logos have been commonl emploed in he elevision broadcasing indusr. The serve as a means of digial righs managemen in he sense ha he are a direc declaraion of he ownership of he video conen. Almos all sicoms news programs and spors broadcas have a logo in a corner of he video frames. Even when he conen has been bough b one channel from anoher he logos are reained and re-broadcased. Therefore he videos ofen have muliple overlapped logos in he same place. While i serves he broadcasers inenion of announcing he ownership i degrades he viewing eperience of he elevision viewer because of he consan obscuring of a par of he conen. Thus i would be ineresing o see how such logos could be removed. emoval of logos can also be consrued o be a form of an aack on visible waermarks [16] [18]. a b Fig. 1. Video logos in he red recangles. a The overlapped logo. b The blended logo A video logo can be of wo disinc sles: he overlapped pe and he blended pe. A direcl overwrien logo overlaps a porion conen of video frames in a manner ha compleel obscures he underling conens of he video such as he logo locaed a he boom-righ of Fig. 1a; while a ransparenl overwrien logo blends iself wih he hos media for eample he logo ha lies a he boom of Fig. 1b. The blended logo allows he conen of he visual media o remain pariall visible. 2
2 elaed work The process of removing a video logo can be hough of as an aack on a visible waermark which is o erase he embedded waermark informaion from is hos media. Mos invisible waermarking schemes can be defeaed b aack sofware such as Sirmark Checkmark and Opimark [32]. I is herefore no surprising ha visible waermarks in form of video logos are sill popular worldwide and he have proven o be a fairl robus e inepensive form of digial righs proecion mechanism. Unforunael video logos affec our viewing pleasure precisel due o heir robusness. Our moivaion for his work was o develop a echnique for removing such logos in order o resore he original viewing eperience. The easies and he mos sraighforward mehod of removing video logos would be o crop ou ha porion of he frame. However man video logos occup a significan area of video frames so ha he cropped video frames lose a ver large amoun of visual informaion. Image inpaining is an ineresing echnique which poeniall could help in he removal of video logos [2][3]. The erm inpaining an arisic snonm for image inerpolaion refers o he echnique of filling in of an gaps in phoographs. The goal of inpaining is o resore he missing or he damaged porions of he phoos. In [2] he auhors borrow ideas from classical fluid dnamics o propagae isophoe lines coninuousl from he eerior ino he inerior of he region o be inpained. The echnique allows he users o fill-in numerous regions conaining compleel differen srucures and surrounding backgrounds. This works ver well for image inpaining hough i is an involved echnique ha is no eas o implemen. Several oher works have since followed. One is based on he erapolaion of neighboring piels recover of edges and curvaure-driven diffusion [5]. A fas inpaining algorihm based on isoropic diffusion model eended wih he noion of user-provided diffusion barriers was proposed in [19]. A muliresoluion image inpaining echnique was provided in [27] based on inpaining a differen levels bu he muliple laer work appears no be ver efficien. Chan e al. have invesigaed he inpaining echniques o come up wih he general mahemaical and variaional descripions 3
[6][7]. In [7] he presen a heoreical analsis for a BV bounded variaion image model which is used for image resoraion; in [25] fundamenal problems in image inpaining and image processing are discussed and some heoreical conribuions have been made owards solving his problem. Image inpaining also has been applied owards aacking of visuall waermarked images in [1] and [11]; image inpaining of muliple views is proposed in [14] and [3] presens an algorihm for he simulaneous filling-in of eure and srucure in regions of missing image informaion. a b Fig. 2. Image inpaining. a An image wih a logo XINHUA. b The inpained image We had earlier developed an algorihm based on image inpaining o erase video logos [31]. We manuall selec he logo region and choose he cleares logo among all video frames. Since ever frame has he logo some have a clear appearance while ohers have he logo mied wih he background. We selec he cleares logo in order o precisel mark he region. We hen auomaicall erase he logo based on region filling b erapolaion. Figure 2 provides an eample o illusrae he removal of logos based on image inpaining. However i is onl a parial soluion o he video logo removal problem because our inpaining echnique canno handle large gaps lef due o overlapped logos. We herefore have developed a new echnique of video inpaining based on eure snhesis. The proposed mehod works effecivel on boh overlapped logos and blended logos. Bornard e al. [4] have also done relevan work on missing daa correcion in an image sequence including logo removal. Alhough he algorihm provided in [4] has he addiional benefi of avoiding he rick sep of moion vecor repair and of smoohl adaping o he moion 4
when processing video i canno auomaicall deec he logo and remove i from he moion picures. Hence manual inervenion is required. The issue of logo deecion has been considered in he domain of documen undersanding. In [24] a se of grascale feaures is eraced o consruc a suie of rules for classifing he segmenaion of he logos. In [28] a mehod o mach logos based on posiive and negaive shape feaures is proposed. In [26] conour-rees and line adjacen graphs are adoped for logo deecion; recursive neural neworks have also been applied o classif logos. The use of grascale feaures prevens hese mehods from eploiing he unique characerisics of video logos. B uilizing color informaion we differ from he pas work. We erac feaures from boh he luminance and chrominance componens since our observaions indicae ha he disincive feaures of video logos are heir shapes and colors. Moreover we use a neural nework o deec he locaion of video logos. Our goal is o deec pical logos from video frames: overlapped logos and blended logos. Fig. 3. Flowchar for video logo deecion and removal In his paper we describe boh video logo deecion and is subsequen removal. For he video logo deecion we presen wo approaches: one is based on video moion and anoher is based on a Baesian classifier. For video logo removal we provide wo soluions: one is o overlap a logo 5
region b seeking he bes mached frame region and anoher is o remove he video logo b using inpaining. As shown in Fig. 3 afer a video clip is inpu we deec a video logo b using a frame differencing based approach. If ha fails we use a Baesian classifier framework in combinaion wih a neural nework based approach and prior knowledge o deec video logos wih he aid of a logo daabase. Afer we obain he recangular region of he video logo we refine i o obain he logo ouline. Once he precise logo region is obained we remove he logos b using eiher a maching based overlapping echnique or a video inpaining echnique. The reason for using wo differen algorihms is ha disinc approaches are required based on he logo background condiions. For insance logo deecion based on frame difference compuaion has he capabili o deec he logos on frames wih moion bu i canno deec hem on saic frames. In order o deec logos under realisic saic condiions we emplo a logo daabase in order o search for hem b using a Baesian approach. In order o improve he accurac of deecion we assume ha he probabiliies of logo appearing a he four corners of he video frames are higher han a he cener. We combine his prior knowledge wih a neural nework based local feaure classifier. The primar reason for using wo approaches o erase logos from video frames is ha he resul of maching based overlapping approach is no saisfacor if he moion of he logo region is insufficien o epose he region underneah. Anoher reason is ha if he logo region is oo large he overlapping will resul in observable edges for he video region. Hence he video inpaining approach provides an effecive alernaive for logo erasure. The paper is srucured as follows: we provide our echnique of logo deecion in secion 3 and removal in secion 4; our resuls will be described in secion 5; he conclusion and fuure work will be cied in secion 6. 6
3 Deecion of he video logo If a video logo is o be removed he firs sep is o locae he posiion of a video logo. In [31] we had o manuall selec he region of he video logo firs and hen seleced he bes logo among all he frames. However his is no pracical herefore we have developed a new mehod o auomaicall locae he video logo in a frame. 3.1 Iniial locaion of he logo Video logo as a percepual waermark for digial videos has some feaures such as disincive shape and visible colors. I also has a sable posiion for a long duraion wihin a video so ha i can draw he aenion of viewers and hus can be idenified as a rademark [9]. Suppose a video wih a logo is denoed b V=v ijk L W H where v ijk is one of he color channels L is he oal number of frames in he video W and H are he widh and heigh of he video frames respecivel. We also need a Boolean arra A=a mn W H a mn =True or False. During iniializaion a mn =True m= W-1; n=1 H-1 and he color disance of corresponding piels beween wo neighboring frames is given b he disance: d ijk = v i1jk - v ijk i=1 L-1. 1 A hreshold ε deermined empiricall o measure he color disance is inroduced o updae all he Boolean values in he arra: If d ijk >ε hen a jk =False else a jk =True. The advanage of his simple approach is effecive a he earl sage o filer ou a large porion of piels ha belong o he moving objecs and background. I allows us o furher develop a more precise logo classifier o refine he resuls. 3.2 Baesian approach for logo deecion In his secion we propose a Baesian classifier framework o perform video logo deecion ask. 7
a b c Fig. 4. egion based deecion approach. a A sample frame wih a logo. b A sample frame wih logo-les deeced. c A sample frame wih a logo deeced. We use local feaures insead of global feaures due o he following observaions on he feaures of he video logos: 1. Color: Good logos are designed o allow he viewer o easil remember and idenif i. The colors of a logo are disincive even if he logo is mied wih a comple background. Thus primar colors such as brigh red green or blue are ofen used. This observaion suggess he use of full color feaures GB o perform he deecion ask. 2. Shape: The shape of each logo is unique and disincive which obviousl disinguishes i from is surroundings. I is worh noing ha while here is a lo of variaion in he overall shape of logos he local shape eure appears o be similar. As shown in Fig. 5 alhough he hree logos are ver differen round recangle and square respecivel he local shapes are similar wih verical horizonal and diagonal edges. For paern classificaion he good feaures of classificaion should have fewer variaions wihin a class and more variaions among classes. Thus he feaures o be seleced should be relaed o he local shape informaion and no he global shape informaion. This suggess he use of local feaures which reain he local shapes while being oleran o he variaions of he global shapes. In addiion deecion ask performed b using local feaures is oleran o he variaions of he size of he video logos as well. 8
a b c Fig. 5. Logos wih differen global shape and size bu wih same local shapes. a ound shape. b ecangle shape. c Square shape We herefore inroduce he noion of a logo-le. A logo-le is a fragmen of a logo obained b breaking a logo region ino 12 12 piel-sized fracions. We erac he GB color informaion from he logo-les. Since he feaures capure he local shape eure informaion in boh luminance and chrominance componens of he logos he allow us o deec and segmen he logo-les from he video regardless of he disinc global shapes and sizes. As shown in he Fig. 6 for each piel of he 12 12 piel-sized fracion GB color values are eraced o consruc he feaure vecor F a oal of 12 12 3 elemens in one feaure vecor. Again insead of deecing he logo region we look for he logo-les a firs and hen combine hose found pieces ogeher o complee he enire logo deecion. As shown in Fig. 4 here are wo seps: firs he 12 12 piel-sized logo-les are obained; second he logo is found b final combinaion of he deeced logo-les as shown in Fig. 4c. Fig. 6. Feaures from he logo-les 1 logo-le eracion 12 12 piel sized fracion 2 feaure vecor F eracion GB feaures from 12 12 piel-sized fracions Compared o he global feaures approach here are wo advanages in using local feaures: olerance o boh shape and size. In addiion he combinaion of he deeced logo-les could creae a good approimaion of he logo regions as shown in Fig. 4c. This is quie imporan for 9
he ne sep: logo removal since i needs a good segmenaion of he logo region raher han jus an indicaion of presence/absence. Based on his feaure we propose a novel logo deecion approach. Le F in d d = 432 in our case from GB feaures in 12 12 piel-sized logo-les each dimension has been normalized o he range [ 1] be a random variable sanding for he local feaure vecor eraced from he 12 12 piel-sized region. We denoe r as he label of he region. I has wo underling class labels {1 }: eiher r = 1 if i is a logo-le region or r = if i is a nonlogo-le region. Given he feaure F we wan o find a classificaion C which is o judge wheher he region belongs o nonlogo-le region r = or logo-le region r = 1 class based on he feaure F C: F r. We inroduce a Baesian decision framework o embed our prior knowledge ino he above classificaion ask. 3.2.1 Baesian classifier In conras wih oher objecs i.e. faces cars eising in video scenes logos have a clear posiional preference. The normall have a higher probabili of occurrence a he four corners han a he cener of he frames. Therefore in order o uilize his aribue besides emploing he feaure F we also add he locaion variable ino our framework. We denoe l = as he locaion of he region represened b is cenroid. The classificaion problem becomes: C:F l r which is o classif he region ino nonlogo-le region r = or logo-le region r = 1 b knowing boh l and F. B appling he Baes heorem [8][17] he classifier can be represened b he poserior probabili P r = 1 F l. Tha is given l he locaion of he region and is feaure F which is he probabili ha he region is a logo-le. We can define he classificaion as: C: r 1 Pr = 1 F l >.5 = 2 oherwise B appling he Baesian rule i is eas o obain he following equaion: P r = 1 l F P l = r = 1 F P r P l F = 1 F 1
We assume ha F and l are independen i is normall rue. The equaion becomes: P l r = 1 P r P l = 1 F = kp r = 1 F P l r = 1 3 where k =1/ P l is a consan. Herein he second facor is he probabili P r = 1 F which is he poserior probabili ha he region is a logo-le given is feaure F. The hird facor Pl r = 1 is he likelihood of he region regarding is locaion l i reveals he posiional variaion of video logos on a video frame. Since i normall has a higher probabili a he four corners of he frames han ha a he cener porion we divide each frame ino four blocks and approimae he probabili b a mied Gaussian disribuion. We use four bi-variae Gaussian disribuions o represen his probabili. P l r 4 = 1 = i= 1 2π 1 i e 1 1 l µ i l i 2 µ T i 4 where i is he covariance mari a he corner i and µ = E l ] is he epecaion of he locaion a he corner i i = 1234 represens he four corners: op-lef op-righ boom-righ and boomlef respecivel. Those parameers are obained b raining from he ground ruh of he logo-le locaion. During he raining each frame is divided ino 2 2 blocks for four corners and each of four Gaussian componens N i µ i i i=1234 is rained respecivel according o is corresponding corner. The poserior probabili P r = 1 F is relaed o he local feaure F. In our framework i is epressed b he oupu of a rained neural nework which will be eplained in he following secion. 3.2.2 Poserior probabili using neural neworks Neural neworks have he abili o implicil model he non-linear relaionships. ecenl i has been successfull applied for he deecion problem boh in he piel domain [22 23 26] and he compressed domain [3]. This has moivaed us o use a neural nework o represen he abovemenioned poserior probabili. The oupu of a neural nework [8 17 21] in he range i [ 11
[1] can be pracicall considered o be proporional o he classificaion confidence. Thus we have developed a full conneced hree-laer feed-forward neural nework [29] 432 inpu unis 2 hidden unis and 1 oupu uni based classifier which is rained b he feaure vecor F 432 elemens from a 12 12 piel sized region. The oupu of he neural nework represening he poserior probabili P r = 1 F is shown as below: P r = 1 F O [1] 5 where O is he oupu of he neural nework. We use he back-propagaion algorihm o rain he neural nework. We consruc a daabase our daa has been colleced from he Web which conains 379 logos. The feaure vecors of a logo-le region are eraced from he daabase based on he eracion scheme as shown in Fig. 6. These feaure vecors are emploed o build he logo-le region model which are sored in he form of he parameers in he rained neural nework and hese parameers represen he disribuion of he logo paerns. For he nonlogo-le feaures 48 video frames picked from several movies and adverisemens are emploed. Toall 8573 logo-le region feaures and 28831 nonlogo-le region feaures are uilized for his supervised learning. l F Neural nework Curren frame Scan window Pl r = 1 P r = 1 F P r = 1 l F Mark as Logo-le region es >.5? no Non logo-le region Fig. 7. The deecion rouine for logo-le regions Figure 7 illusraes he flowchar of our logo-le region deecion approach. In Fig. 7 a recangular window wih a fied size of 12 12 piels is used o scan he whole area of a video 12
frames. The feaure vecor F GB values of 12 12 piels oall 432 elemens in each window is eraced. The eraced feaure vecor F is inpu ino he rained neural nework and he oupu is regarded as P r = 1 F. On he oher hand based on he curren locaion in he frame l he probabili Pl r = 1 is obained from Eq. 4. Consequenl b using hese probabiliies he poserior probabili P r = 1 l F is calculaed from Eq. 3 and he curren region is accordingl classified as logo region or nonlogo region based on Eq. 2. Noe ha he oupu of he flowchar of he Fig. 7 is he deecion procedure of he logo-le regions. However afer he logo-le regions are deeced hese regions need o be amalgamaed in order o achieve final oupu. This pos-processing procedure has wo seps: 1 merge all he neighboring log-le regions ogeher 2 remove an of he isolaed log-le regions which are no bigger han 12 12 piels in size. Afer doing his pos-processing he final resuls are obained. 3.3 efining he video logo locaion We call he logo before i is refined o be a coarse logo. I refers o a region which ma no have a clear background. Our ask is o find he logo wih a clear background. Afer marking he posiion of he video logo he region perhaps includes some deails ha are reall no a par of he logo informaion. A locaed logo ha has a lo of background cluer is of a low quali. We need refine his iniiall deeced logo b calculaing he logo edge lenghs. Suppose a coarse video logo is f =1 m-1; =1 n-1 he ober edge deecor is uilized o erac he edges in he region [29]: f 2 2 = f 1 f f 1 f 6 A hreshold E T is used o decide wheher a piel is on he edge or no. Therefore he lengh of he edges in logo becomes a disinguishing feaure. Given ha he coarse logo is deeced in all frames of a given video we wan o pick he frame having he cleares logo. We selec he logo region wih he shores lengh of edges as he bes logo frame. This is because a bad logo is alwas accompanied b a comple background eure; on he oher hand a good quali video 13
logo displas a clear background wih disinc conras wihou addiional eure obscuring he logo area. eling on his observed feaure we refine he obained logo. The logo refined b his mehod is ver close o he real logo if he noise edges are ver shor. The use of his algorihm will be clear wih an eample. a b Fig. 8. Logos of differing quali. a A good quali logo. b A bad quali logo Figure 8a and Fig. 8b illusrae wo logos of differen quali in wo frames. The logo in Fig. 8a is clear he background of he logo is green grass of soccer ground and i has less eures and edges. The logo in Fig. 8b has a comple background; he logo is blended wih an adverisemen board of adidas such ha we canno disinguish he conen of he logo clearl. We canno easil deermine wheher adidas is a porion of he logo or no. 4 emoval of he video logo We could have considered using he image inpaining echnique for logo removal. However since his mehod canno erase logos having a large area we adop a differen approach in his paper. We remove he logo b obaining he mos similar region in video sho o fill he region. I is akin o he idea of moion compensaion used in compression albei wih an idea of doing region resoraion. Alhough man disance measures can be seleced for maching herein we adop he maching mehod based on he Hausdorff disance for he sake of simplici. The advanage of our proposal is ha i is auomaic wihou human inervenion. Moreover we develop he idea of video inpaining. In his paper we imagine he logo in a frame sequence o be 14
a clinder and our job is o fill he logo clinder wih he help of neighboring frames and boundar informaion his will be illusraed in laer Fig.1. 4.1 Maching based logo removal 4.1.1 Maching he logo region based on he Hausdorff disance Given wo finie poin ses A={a 1 a 2 a m } and B={b 1 b 2 b m } m> is an ineger he Hausdorff disance is defined as: hab=ma{hab hba} h A B = ma min a b a i A a A b B i j b j B i j=1 m; is a norm on he poins of A and B. The ke advanage of using he Hausdorff disance o compare he difference of images is is low sensiivi o small perurbaions of he image. The compuaion of Hausdorff disance differs from man oher shape comparison based mehods in which here is no correspondence beween he model and he image. The mehod is quie oleran of small posiion errors such as hose ha occur wih edge deecors and oher feaure eracion mehods [1 12 13 2]. Alhough we do no use he piel posiion informaion in he compuaion of Hausdorff disance each piel has is fied posiion in he blocks. Here we are mainl concerned abou he posiion of each search block insead of he posiion of each piel. Assume ha a video logo occupies a porion of video frames Ω w we need find he mos similar region Ω s among all video frames v ijk L under he definiion of Hausdorff disance. The conen of his region is he mos similar o he conen of he region ha he logo occupies Ω o. Namel: d c = h Ω Ω d s = min{ d c } o c Ω = Ω 7 s c d c = d s If we can find his region i can be used o replace he logo. Bu his mehod has a problem since he logo overlaps he region we canno find he bes mach for i. Because we do no know wha lies underneah he logo adjacen regions are considered. The regions o he righ lef op and boom of he video logo region are considered for maching wih oher frames in he sho. 15
For he curren frame we mach hese four regions wih all he oher video frames. We find he mos similar pair among all he candidae pairs in he video frames. The corresponding posiion of video logo in he bes-mached frame will be found. For insance if he highes similar pair among he candidae regions is on he op of logo region in a video he region under he highes similar region of he mached frame will replace he logo region. Subsequenl he region wih he logo will be replaced and hus he video logo is removed. 4.1.2 Maching based logo region overlapping When his algorihm is uilized o remove video logo a leas hree frames are considered; he are he curren frame ne frame and previous frame in he same sho. Fig. 9. Maching based algorihm for region overlapping In Fig. 9 suppose he middle frame is a frame of he inpu video wih a logo. Around he video logo four regions are marked wih A B C and D. In order o obain he subsiuion region of he logo he regions in he relaed frames previous frames and ne frames of he curren frame are considered wih he mos similar pair among all he candidae pairs being seleced. The corresponding region is hen used o replace he waermarked region. If he mos similar pair is on he op of he logo he region under maching is uilized o overlap he visible waermark shown in he lef par of Fig. 9; if he mos similar pair is a he boom of he logo he area above he mached region is uilized o overlap he logo as shown in he righ par of Fig. 9. Since he objecs in video are moving he region occupied b he video logo in he curren frame ma no be obscured b he logo forever. Thus we have he opporuni o locae ha region eposed which can be subsequenl used for removing he logo from he curren frame. 16
The drawback of his echnique is similar o ha of locaing he logo region. If mos objecs in he frames of arge video are saionar we perhaps canno find a beer subsiuion region o replace he region occupied b he video logo. However we are surel able o find some regions o replace he logo even if i is no he opimal one. Anoher shorcoming of he proposal is ha someimes he found blocks will bring era visible edges afer overlapping and desro he coherence of his region. Therefore we would like o inpain he video frames using video inpaining. In shor overlapping based logo removal is a supplemenar measure o handle he logos wih moion in he background for a leas one logo region posiion. If he moion is no significan video inpaining can be emploed o fill he logo region. 4.2 Video inpaining based logo removal Unlike image inpaining and block based overlapping algorihms for video logo removal video inpaining considers a frame sequence o fill a logo region. The logo region o be filled in he frame sequence is reaed as a clinder. The processing of video logo removal can be considered as he filling of he logo clinder in he video volume laer from he ouermos o he innermos laers. Our scheme is illusraed in Fig. 1. Three eample frames of video are shown in he diagram. The recangles represen piels in frames wih he red ones indicaing he curren piels o be inpained. In Fig. 1b he green recangles depic piels ha are inpained laer and he informaion of he blue piels is uilized o inpain he curren piels. The ellow piels are also uilized in he rendering of he curren ones. 17
a b c Fig. 1. Video inpaining based logo removal. a clinder laer filling based video logo removal. b Each piel on he laer of clinder is inpained b using adjacen laers. c The vecors relevan o logo edges In Fig. 1 we overlap he logo region b using he 3D gradien vecors insead of 2D gradien vecors normall used in image inpaining o obain a region o replace he logo. We will compue he conribuions of he gradien vecors of adjacen frames and predic he color value of he curren piels. We also use moion vecors of logo region in order o find he piels on he ouside rings. The idea is o fill he marked region more precisel han he mehods proposed earlier. Suppose Ω is he logo region wih Ω being is boundar. A piel on he boundar Ω ha needs o be inpained is denoed as z = f where is he piel posiion is he frame on which i is locaed. The angen vecor is Tf f f and he normal vecor Nn n n lies on a plane where T N=. The gradien vecor G-n -n -n is also on his plane. In order o inpain he logo clinder we onl uilize hose normal vecors II I I whose direcion poins owards he ais of he clinder namel: G 1 T > as shown in Fig. 1c. We esimae he inpained piels b using he firs order difference: f = f1 - f 8 f = f 1 - f 9 f = f 1 - f 1 f = w 2 f-1 - f-2 w 2 f-1 - f-2 w 2 f-1 - f-2 11 18
where he w [1] w [1] and w [1] are weighs ha indicae he conribuion of he various gradiens o he curren piels and w w w =1. In emporal domain we eploi video frame coherence and uilize he correlaion beween hese frames. Thus we need o compue he 3D gradiens from differen direcions. No onl do we use he 2D gradiens o compue he influence on he curren piels; we also uilize he emporal domain informaion from adjacen frames. We compue hese gradiens o inerpolae he piels on he curren frame. The video inpaining based mehod fills he concenric clinder sep b sep along boh he emporal and spaial direcions b propagaing he region informaion. The video inpaining processing can be visualized as he dissolving surface of a clindrical ice block in a ank of waer. One of challenges of his 3D dissoluion mehod is ha we need o obain he moion esimaion; we use his o infer he opical flows and isohes. 4.3 Issues in video inpaining In his secion we will discuss some heoreical issues. One open issue is how large a logo area can be inpained in videos and he oher issue is how o pach he cracks appearing due o he overlapping processing in order o mainain coherence beween adjacen video frames. Herein we briefl inroduce he ouline of he approach. 4.3.1 The area of video inpaining Basicall a region of an size ha a video logo occupies can be inpained b using our proposal. However he quali of he inpained video and is correlaion wih he video scene will be a funcion of he logo size and shape. Thus in his secion we will discuss he relaionship beween he region size and he quali of he inpained video. The fideli of video inpaining is subjec o he disribuion and densi of marked piels in he inpained region. Figure 11a provides an ideal disribuion he black blocks represen he marked piels while he whie blocks represen he original ones. In his case for four-conneced region here are four whie piels around each black piel; and here are four black piels around each whie piel. Thus each black piel o be inpained is able o infer he piel value from is 19
adjacen piels. Consequenl he area o be inpained wih a high fideli can be as large as possible. Namel he inpained area achieves he maimum value of he quali no maer how large he area of marked region is because here is hardl an error in he inerpolaion. Thus he fideli of inpained video will be beer if he marked piels are fewer han in he case menioned in Fig. 11a while he disribuion remains he same. Or else if he densi of he marked piels is high while reaining he disribuion he resored quali will be degraded. Figure 11b presens he wors case for video inpaining fideli relaed o inpaining densi and disribuion. Cerainl if he marked region is enirel filled b he marked piels he fideli of he resored region is he lowes. Errors will be propagaed over a large disance in his case. a b Fig. 11. Disribuion and densi of marked piels in video inpaining. a The bes-case disribuion of marked piels. b The wors-case disribuion The logo is a smbol ha covers a porion of video frames. Video inpaining is he approimaion of he covered region b using he surrounding informaion and heir differences. Thus he region o be filled is a hole or a clinder which has emporal and spaial coherence. For logo removal he conribuions of each ring surrounding he logo region four-conneced region should saisf Eq. 11. We now esimae he conribuions of each ring piel o he boundar region: f = w -1 f-1 f -1 w -1 f -1 f -1 w -1 f -1 f -1 12 The error is: = f - f = w -1 f - f -1 w -1 f - f -1 w -1 f - f -1 = V 1 -V 2 13 where V 1 = w -1 f w -1 f w -1 f T and V 2 = w -1 f -1 w -1 f -1 w -1 f -1 T. Equaion 13 suggess ha he inpained piels are closel relaed o he gradien vecors of neighboring rings. 2
21 If we use he gradien of onl he piels near he edges o esimae he piels far from he edges his will lead o error. If a piel o be inpained is far from he edge he piel value is less accurae. Thus i is impossible o inpain an infinie een of a region. Acuall he above siuaion can be described beer b aking a coninuous funcion f ino accoun insead of he discree epression. Then is Talor series epansion a > > > is: ρ o f f f f f = 14 where 2 2 2 ρ = 15 In general if he video frame is represened b a coninuous funcion f wih n1 order parial derivaives in he region Ω={ > > >} Ω hen he Talor epansion of he funcion a is given b: n n f n f f f f ε =! 1 2! 1 2 L 16 where m c b a f C f m r c b a m c b a r m m = = = 17 ε n is he remainder: 1 1! 1 1 = θ θ θ θ ε n s n f n 18 ε n in 18 gives he esimae of he error for a coninuous funcion in image inpaining. 4.3.2 Siching of cracks in eure overlapping In general a suiable block can alwas be found o overlap a logo region no maer how large he size of he logo region is and how inferior he quali of overlapped image is. However he
region someimes leaves he race of overlapping and causes percepual quali degradaion. In order o properl remove he obvious edges arising due o overlapping we can use he idea relaed o eure cus [15]. In [15] video eures can be easil cu b an inelligen scissor; blending auomaicall sews he seams of wo overlapping frames. Unless here are unsuiable significan or obvious lighing and color variaions he overlapping edges are undeecable. Thus a reasonable soluion would be o find a similar block and blend he edges [15] so ha he edges are no visible. 5 esuls In his secion we describe some of he eperimenal resuls. The online version of hese eperimenal resuls can be found on our websie [33]. 5.1 Logo deecion 5.1.1 Logo deecion based on frame difference Figure 12 liss he resuls of video logo deecion Fig. 12a is he original video frames wih a logo Fig. 12b is he video frames wih deeced video logos. From he resuls we can clearl see he video logo is successfull deeced. In his eperimen he hreshold for logo deecion is 64. If he sum of he difference beween wo frames is greaer han 64 we assume ha moion eiss. a b Fig. 12. Video logo deecion based on frame difference. a The original video. b The video logo is deeced 22
5.1.2 Logo deecion based on neural nework In order o demonsrae our proposal we esed several video clips colleced from various websies. The deecion rae obained is over 82% on 236 frames 1 non-logo frames 226 frames wih logos eraced from 23 differen video clips conaining 23 differen video logos lised in Table 1. Table 1. Video logo deecion resuls Toal logos 226 Deeced logos 187 Missed logos 39 False alarm 36 Deecion rae 82.7% Our eperimens show ha our approach for video logo deecion is quie robus across various pes of logos in erm of shape and size. Figure 13 shows some sacked TV logos and Fig. 14 illusraes some blended logos from various websies. I indicaes ha he local feaures we used are oleran in shape and size due o he fac ha he are eraced from a small 12 12 sized regions. a b c Fig. 13. Deeced resuls of overlapped logos. a TV logo 1. b TV logo 2. c TV logo 3 a b Fig. 14. Deeced logos of blended logos. a Blended logo 1. b Blended logo 2 23
5.2 Logo removal 5.2.1 Video logo removal based on overlapping a b Fig. 15. Logo removal. a An original frame of a video wih logo. b The corresponding frame in video wih removed logo PSN = 25.7dB Figure 15a is a frame of video wih a logo on is op-righ corner and Fig. 15b is he corresponding frame wih he logo removed. We remove he logo b overlapping a similar region found in he video sho. The recangle on he frames indicaes he posiions of he logo. We ake a video wihou an logo NUS Campus shown in Fig. 16a and add five differen logos a differen posiions of he video frames. The logos used are shown in Fig. 16b. We remove hese logos b he maching based overlapping approach and obain five resored videos wihou logos. Some resulan frames are lised in Fig. 16c he logo-removed regions are marked b recangles such as op-righ boom-righ boom-lef op-lef corners and cener ec. We hen compare he original video and he resored videos wihou logos b compuing he difference beween each pair and show he PSN Peak Signal o Noise aio values in db obained as resuls. The PSN values are shown in Fig. 16d. a 24
b c d Fig. 16. Video logo removal based on overlapping. a Original video wihou logo. b Five logos for eperimenal resul. c Some good overlapping frames. d PSN comparison in 5 differen logo-erased videos From Fig. 16 we can see ha he eperimenal resuls are quie good. The average values of PSN are around 35.5dB or so. I is well known ha if he PSN values are greaer han 3dB he wo compared images are percepuall similar. Maching has been used o overlap he logo region for dnamic videos. Since i does no work well for a saic video we use video inpaining for such cases. 25
5.2.2 Video logo removal based on video inpaining In case he direc overlapping mehod does no work we aemp o remove he logo using video inpaining. Figure 17 shows he eperimenal resuls for his case. We erase he green logo in he video shown in Fig. 17a and obain he video shown in Fig. 17b. Figure 17c is he comparison beween he original video and video wih erased logo. We can see he PSN value in Fig. 17c is above 36dB which indicaes ha we almos canno find an percepual difference. Alhough he video is same as in Fig. 16 he resul is differen since we use a differen mehod o inpain he video logo. a b c 26
Fig. 17. Logo removal based on video inpaining. a Video wih marked logo. b Video wih erased logo. c Comparison beween he original video and video wih erased logo Figure 18 liss one of resuls when he echniques are used in real video logo handling. I is a piece of news video from he Iraq war. Some frames wih he logo of U-Channel in Singapore are provided in Fig. 18a and some frames wih he logo removed are shown in Fig. 18b. Figure 18c and Fig. 18d show resuls of anoher eperimen: Fig. 18c is he animaion of he Iranian win sisers conneced skull wih he logo 11 on he frames and Fig. 18d is he animaion wih he logo removed. a b c d Fig. 18. Logo removal in various video clips. a Iraq war video wih a logo. b The video a afer removed logo. c Iranian win sisers conneced skull wih he logo. d The video c afer removed logo Figure 19 illusraes a resul when he echniques are used o remove he capions on a karaoke video. Figure 19a provides a karaoke clip wih capions while Fig. 19b presens he corresponding clip wih removed capions. 27
a b Fig. 19. emoval of capions on a karaoke video. a A karaoke clip wih capions. b The clip wih removed capions We have shown he resuls of logo deecion and removal as well as for real logo handling in a news video clip and a karaoke clip. From hese resuls we can see ha he proposed echniques are quie effecive. 6 Conclusion Video logo removal is a hard problem despie is apparen ease. We have developed a new echnique based on frame difference and neural neworks o locae he video logo posiion. Moreover a logo-refining algorihm has been provided for obaining a more precise logo region. In case ha we fail o obain a good mach he echnique of video inpaining can be uilized. The eperimenal resuls indicae he feasibili and effeciveness of our echnique. For fuure work we will consider he aenion-based and cone-based logo deecion on fabric. We will also aemp online logo removal from live video sreams. Anoher challenge would be o erase logos direcl in compressed-domain MPEG videos. eferences 1. Beloga E Cabrelli C Moler U Shonkwiler 1997 Calculaing he Hausdorff disance beween curves. Informaion Processing Leers 64: 17 22 28
2. Beralmio M Sapiro G Caselles V Balleser C 2 Image inpaining. In: Proceedings of ACM Siggraph 2 pp 417 424 3. Beralmio M Vese L Sapiro G Osher S 23 Simulaneous eure and srucure image inpaining. IEEE Transacion on Image Processing 128: 882-889 4. Bornard Lecan E Laborelli L Cheno J 22 Missing daa correcion in sill images and image sequences. In: Proceedings of ACM Mulimedia 22 Juan Les Pins France pp 355-361 5. Chan TF Shen JH 21 Non-eure inpaining b curvaure-driven diffusions. Journal of Visual Communicaion and Image epresenaion 124:436 449 6. Chan TF Shen JH 21 Mahemaical models for local deerminisic inpainings. SIAM Journal Applicaion Mahemaics 623:119 143 7. Chan TF Shen JH 23 On he role of he BV image model in image resoraion. Amer. Mah. Soc. Conemp. Mah. 33: 25 41 8. Duda O Har PE Sork DG 2 Paern Classificaion. Wile Inerscience 2 9. Holland J Hanjalic A 23 Logo recogniion in video sills b sring maching. In: Proceedings of IEEE Inernaional Conference on Image Processing ICIP 23 Barcelona 23 1. Huang CH Wu JL 21 Inpaining aacks agains visible waermarking schemes. In: Proceedings of SPIE 4314 pp 376 384 11. Huang CH Wu JL 24 Aacking visible waermarking schemes. IEEE Transacions on Mulimedia 61: 16 3 12. Huenlocher D Klanderman G ucklidge W 1998 Comparing image using he Hausdorff disance. IEEE Transacion on Paern Analsis and Machine Inelligence PAMI 159: 85-863 13. Huenlocher P Lilien H Olsen F 1996 Approimae Hausdorff maching using eigenspaces. In: Proceedings of he APA Image Undersanding Workshop pp 1181 1186 29
14. Kang SH Chan TF Soao S 22 Landmark based inpaining from muliple views. In: Proceedings of Inernaional Smposium on 3D Daa Processing Visualizaion and Transmission 3DPVT June 22 15. Kwara V Schödl A Essa I Turk Gm Bobick A 23 Graphcu eures: image and video snhesis using graph cus. ACM Transacions on Graphics 223: 277 286 ACM Siggraph 23 16. Meng J Chang SF 1998 Embedding visible video waermarking in he compressed domain. In: Proceedings of IEEE Inernaional Conference on Image Processing ICIP 1998 Chicago pp 474 477 17. Michell TM 1997 Machine Learning. McGraw-Hill 1997 18. Mohan SP Kankanhalli MS amakrishnan K 2 A DCT domain visible waermarking echnique for images. In: Proceedings of IEEE Inernaional Conference on Mulimedia and Epo ICME 2 New York pp 129 132 19. Oliveira MM Bowen B McKenna Chang YS 21 Fas digial image inpaining. In: Proceedings of he Inernaional Conference on Visualizaion Imaging and Image processing VIIP 21 pp 261 266 2. Olsen F 1998 A probabilisic formulaion for Hausdorff maching. In: Proceedings of IEEE Inernaional Conference on Compuer Vision and Paern ecogniion CVP 1998 pp 15 156 21. Papageorgiou P Oren M Poggio T 1998 A general framework for objec deecion. In: Proceedings of Inernaional Conference on Compuer Vision ICCV 1998 Mumbai India pp 555 562 22. owle HA Baluja S and Kanade T 1998 Neural Nework-Based Face Deecion. IEEE Transacions on Paern Analsis and Machine Inelligence PAMI 21: 23 38 23. owle HA Baluja S and Kanade T 1998 oaion Invarian Neural Nework-Based Face Deecion In: Proceedings of IEEE Inernaional Conference on Compuer Vision and Paern ecogniion CVP 1998 pp 38 44 3
24. Seiden S Dillencour M Irani S 1997 Logo deecion in documen images. In: Proceedings of he Inernaional Conference on Imaging Science Ssems and Technolog pp 446 449 25. Shen JH 23 Inpaining and he fundamenal problem of image processing. SIAM News 365 26. Sheng J 1998 Graphical iem recogniion using neural neworks. Ph.D Thesis Universia degli Sudi Di Firenze 27. Shih TK Lu LC Chang C 23 Muli-resoluion image inpaining. In: Proceedings of IEEE Inernaional Conference on Mulimedia and Epo ICME 23 Balimore Marland Washingon D.C. 28. Soffer A Same H 1998 Using negaive shape feaures for logo similari maching. In: Proceedings of he 14h Inernaional Conference on Paern ecogniion ICP 1998 Brisbane Ausralia pp 571 573 29. Sonka M Hlavac V Bole 1999 Image Processing Analsis and Machine Vision 2 nd Ediion. Brooks/Cole Publishing Chapman & Hall Compuing London 3. Wang J Kankanhalli MS Mulhem P and Abdulredha HH 22 Face deecion using DCT coefficiens in MPEG video In: Proceedings of Inernaional Workshop on Advanced Imaging Technolog IWAIT 22 pp 66 7 31. Yan WQ Kankanhalli MS 22 Erasing video logos based on image inpaining. In: Proceedings of IEEE Inernaional Conference on Mulimedia and Epo ICME 22 Lausanne Swizerland pp 521 524 32. hp://www.waermarkingworld.org 33. hp://www.comp.nus.edu.sg/~anwq/acmmmsys/logo/inde.hm 31