Proceedigs of the IASTED Iteratioal Coferece Iteret ad Multimedia Systems ad Applicatios October 18-21, 1999 Nassau, Bahamas Iteret supported Aalysis of MPEG Compressed Newsfeeds Guido FALKEMEIER, Gerhard R. JOUBERT ad Odej KAO Techical Uiversity of Clausthal, Departmet of Computer Sciece, Julius-Albert-Str. 4, 38678 Clausthal-Zellerfeld, Germay Email: {falemeier, joubert, oao}@iformati.tu-clausthal.de Abstract 1 : I this paper ew ideas ad algorithms for adaptig the worflow for ewscast productio are preseted. The goal is to eable TV ews editors to tae advatage of the added facilities offered by digital video techiques. Algorithms for the aalysis of MPEG compressed videos are preseted. These eable a ews editor to extract sequeces of ey frames from ewsfeeds. Newsfeeds cosist of assembled ews clips trasmitted by ews agecies. Key frames are extracted i order to create a quic overview of the available material. I additio frames cotaiig textual descriptios of the ews cotet of idividual clips are also extracted. These images are of low quality ad stadard OCR methods prove usuitable for text recogitio. Usig stadard image processig techiques i combiatio with a ew OCR method especially developed for this purpose ey words are extracted from these frames. O the oe had the ey words ca be used by ews editors to search for particular ews items i a ewsfeed ad o the other to search for related iformatio o the iteret. The fial output of the system thus supplies ews editors with a overview of ewsfeed cotet combied with additioal iformatio gathered from ews agecies via the iteret. Key words: Newsfeeds, Aalysis, MPEG, OCR, Iteret 1. Itroductio I this paper a etwor based support system for TV ews editors is preseted. It combies ew digital image ad digital video processig methods with iteret based iformatio gatherig techiques. Televisio statios receive daily ewsfeeds from various ews agecies, such as Reuters, APTN or CNN. Each ewsfeed cotais curret material coverig may differet ews items, such as impeachmet hearig of Presidet Clito, sow storms i Caada, poiso gas igrediets i crashed aircraft, etc. Each statio aalyses ewsfeed cotet ad selects particular items for their idividual ewscasts. Newsfeed material is curretly copied i broadcast quality from satellite trasmissios to digital video tapes. Copies of these are the scaed by ews editors i order to select material for ewscasts. Selected items are subsequetly copied ad assembled. This process is 1 This wor is part of a joit research project with a commercial televisio statio. labour itesive ad requires specialised ad expesive equipmet. The icreasig volume of material to be scaed ad the limited umber of professioal worstatios available maes it difficult for editors to execute their tas satisfactorily. A urget eed thus exists to ivestigate alterative computer based methods for aalysig ewsfeed cotet. Due to the huge data volumes created whe videos are digitised the storage, commuicatio ad processig of the videos o PCbased computer etwors is ot possible without compressig the material. I the case of the wor cosidered i this paper compressio accordig to the MPEG-1 stadard is used. The quality of MPEG-1 compressed videos is, however, ot suitable for broadcastig by TV statios. The actual ewscast must thus still be assembled from broadcast quality digital tapes. These are recorded i parallel with the creatio of the MPEG compressed versios of the ewsfeeds. With the aid of the ew system the recorded, digitised ad compressed video material ca be aalysed. As a first step the separate ews items must be detected. This eables a ews editor to select particular items without viewig all the iterveig material. This results i a substatial reductio i time eeded for the selectio of items. It also obviates the eed for decompressig the video material. This greatly reduces storage, compute ad commuicatio requiremets of the etwor. Durig the video aalysis phase additioal iformatio, such as ey frames ad textual iformatio o idividual items, is extracted from the compressed video material. All methods used wor directly o the compressed material. This results i high commuicatio ad processig speeds. The method used is described i the first part of this paper. Durig the secod phase the frames recogised as cotaiig textual iformatio about idividual ews items are processed. Digital image processig methods are used to pre-process these. A specially developed text recogitio method the extracts ey words relatig to the ews cotet of the subsequet item. Refer to the secod part of the paper for a descriptio of this process. Durig the fial processig phase a iteret search is executed with the extracted ey words. Tests showed that available iteret search egies are ot suitable for this tas as too much redudat material is geerated. A special search egie was thus developed (see part 3). The 298-120 -1-
result of this search supplies the editor with additioal material relatig to particular ews items. Iformatio from other ews agecies thus becomes directly accessible to him. He ca use this i, for example, preparig the accompayig text of ews items. All iformatio is preseted o the editor s worstatio i a well-structured ad clear fashio by a system based o HTML pages. 2. Detectio of hard scee cuts I films ad videos hard scee chages or cuts are the most commo form of trasitio from oe scee to aother. Soft scee chages where oe scee is gradually faded out ad the ext faded i require special effort ad is rarely used by reporters i ews clips. A scee cut detectio method ca thus cocetrate o the fidig of hard scee cuts. This maes it feasible for such a method to achieve the objectives of beig fast, accurate ad complete. The cut detectio method used here [8-9] is based o a iterative approach cosistig of three phases. The first phase executes a fast global aalysis. This is the refied i subsequet phases to determie the exact cut positio. A cosiderable speed advatage over other approaches is achieved. 2.1 Global aalysis The global aalysis of the first phase cosiders I-frames oly. Two eighbourig I-frames are compared ad, if a scee cut is detected, the method switches to the secod phase. Durig the secod phase oly those P-frames eclosed by two I-frames are aalysed. This results i a more exact positioig of the scee cut. Two eighbourig frames for example a I- ad a P-frame which cotai the scee cut are obtaied. Durig the third phase the icluded B-frames are aalysed. This gives the exact positio of the cut. 1. Phase 2. Phase Figure 1: Multi-phase approach 3. Phase Scee cut Figure 1 shows the phases of the iterative process. The aalysis of the I-frames durig the first phase shows that a scee cut exists betwee the secod ad third I-frames. The secod phase is the etered ad the P-frames betwee these two I-frames are aalysed. This results i the idicatio that a scee cut exists betwee the first ad secod P-frames. Durig the third phase a aalysis of the B-frames determies the exact positio of the cut. The process remais the same irrespective of whether the scee cut is located o a I- or a P-frame. I order to determie a cut located o a P-frame all three phases are executed, with both the first ad secod phases sigallig the existece of a scee cut. The third phase, however, fails to determie the positio of a cut amogst the eclosed B-frames. This meas that two possibilities exist: the scee cut is located o the first P-frame, or the scee cut is located o the secod P-frame. I the first case all B-frames will maily referece the followig (secod) P-frame, as they are all part of the ew scee. O the other had, if the secod case is true, the B- frames will maily referece the first P-frame, as they the form part of the previous scee. Thus, i order to determie at which P-frame the scee cut is located, the refereces of the B-frames must be aalysed. If a scee cut is detected durig the first, but ot the secod, phase the cut must be located o the I-frame or the B-frames followig the last P-frame. The mai advatage of this approach as compared to existig methods is the higher compute speed. This results from the cosiderably smaller umber of frames to be ivestigated. I spite of this there is o additioal loss of iformatio due to the fact that withi a MPEG compressed video approximately every 12th-16th frame is a I-frame. It is highly uliely that at least i the case of ewsfeeds more tha oe scee cut occurs withi this time frame of just over half a secod. 2.2 Scee cuts withi the I-frames The I-frames cotai the complete image iformatio as trasformed by the DCT. The detectio of a scee cut is thus based o the aalysis of the DC compoets of all 8 8 blocs of the itra-coded images. For a video V ad the cotaied I-frames the followig parameters are defied: N The umber of I-frames i V DCI The ordered set of DC compoets of the -th I-frame. The DC compoets are ordered accordig to the sequece of the associated 8x8 blocs i DCI The sequece of all DCI of V The existece of scee cuts is determied by aalysig the sets DCI ad DCI 1 : A, + 1 x + y with x DCI, y DCI + 1, 1,..., N-1 ad the umber of rows ad colums respectively. The result of this aalysis is a sequece of differece images betwee the two sets DCI ad DCI + 1 for all I-frames cotaied i V. A umber of differet orms [5-7] are available for calculatig the differeces. May of these are very compute itesive. The orm proposed here -2-
requires relatively little compute effort. It was tested with success i the case of compressed videos [9-11]. If a scee cut exists betwee two I-frames the calculated distace must be large compared to the average distace betwee I-frames. I practice a scee cut spas a umber of I-frames ad it is thus reasoable to assume that withi a sequece of differece images a large value will be boud by comparatively small values. I the case of a scee cut betwee DCI ad DCI + 1 the followig must hold: N 1 A, + 1 1 A, + 1 > x Aavg, with Aavg N 1, x + l A, + 1 l, { 1,1.1,..., 4} { 1,2,3 }, { 1,1.1,...,4} A, + 1 > y, for l y 2l The values of the parameters x, y ad l are determied through a adaptive correctio [4]. The first coditio detects images, which show up a greater differece tha their eighbourhood. This formula esures a greater robustess tha if the simple maximum orm was used. The secod coditio esures that o additioal large differece images exist i the direct viciity of the frame cosidered. This prevets the false iterpretatio of movemet as scee cuts. 2.3 Scee cuts withi the P- ad B-frames Referecig macro blocs ca be used for the detectio of scee cuts i the case of P-frames i additio to the itracoded blocs. I the referecig macro blocs up to five 8 8 blocs ca be itra-coded. If a scee cut exists betwee the P-frame cosidered ad the referece frame, a large umber of macro blocs will be completely itracoded ad may referecig macro blocs will be partly itra-coded. This results from the big differece betwee the P-frame ad the referece frame. I summary a large umber of 8 8 blocs will be itra-coded. The umber of these blocs ca thus be used as a measure to idicate the existece of a scee cut. I cases where extesive movemets are depicted i a scee this method may brea dow. This is due to the fact that i such situatios the differeces betwee the referece ad the P-frames withi the scee are relatively large. A clear distictio betwee lots of movemet ad scee cuts caot be made. This results i the possible o-detectio of scee cuts withi or borderig o such scees. I order to achieve a higher degree of robustess of the method the average of all differeces, excludig the maximum value, is used. I case of the B-frames it does ot mae sese to base the scee cut detectio o the umber of itra-coded blocs. This is due to the fact that B-frames ca use a precedig or subsequet P- or I- frame withi the same scee as referece. This results i the geeratio of a small umber of itra-coded blocs. Oe possibility to wor aroud this problem is to calculate the differece betwee the umber of forward ad bacward referece blocs for each frame ivestigated. This results i a sequece of umbers i which a sig chage idicates that a scee cut may be icluded i the frame sequece. This is ot a sufficiet coditio as sig chages may also result from roudig errors. I additio to the costructio of the sequece of differeces a secod orm is thus itroduced. For each frame the umber of macro blocs which cotai refereces to both precedig ad subsequet referece frames is determied. This umber is the divided by the total umber of macro blocs i the frame. A small value of this quotiet idicates the existece of a scee cut. The combiatio of these two metrics was show to greatly icrease the accuracy with which scee cuts ca be detected. This algorithm was tested o a umber of ewsfeeds from various ews agecies. I the case of Reuters ewsfeeds, for example, the methods described succeeded i completely segmetig the ewsfeed ito idividual clips. Withi each clip at least 80% of scee cuts were detected. The video stream aalysis delivers a good overview of ewsfeed cotet. A editor easily fids headlies of iterestig ews items. This iformatio is sparse ad the automatic extractio of additioal iformatio from iteral ad exteral archives or the iteret is desired. This ca be achieved through e.g. lis to ow, already broadcasted ews clips, ewspaper articles i o-lie accessible archives, lis to reliable ad up-to-date iteret sites. Such sites are offered ad maitaied by various ews agecies. The extracted iformatio ca be used for further searches to fid bacgroud iformatio ad for the retrieval of related archived ews clips. A importat problem for this cotet based search is the determiatio of iput data for the search egie. I the followig sectio a method for aalysis of cotet frames i the ewsfeeds ad extractio of the icluded ey words is discussed. 3. Key word extractio from cotet frames The textual frames cotai basic iformatio about the followig item i the ewsfeed. Usually this icludes a short title describig clip cotet, duratio, laguage, soud, shot data ad veue. The iformatio is preceded by ey words lie DURATION, SHOT, etc. These are importat for the iterpretatio of the supplied data. The layout of the textual frames differs from oe agecy to the ext. This ecessitates a adaptatio of the extractio method. The process cosists of: 1. Determiatio of the start/ed ad of the textual frames for each group i a ewsfeed, 2. Extractio of the textual frames, 3. Textual frames are prepared by applyig image processig methods for optical character recogitio, 4. Character recogitio ad savig of the ey words. -3-
The first step is to fid the start frames of each group i a ewsfeed. By implemetig a partial decodig of the DC compoets of I-frames, calculatig the histograms ad usig a suitably chose threshold value, blac frames ca be idetified. Further discussio of this method ca be foud i [8]. The frames cotaiig textual iformatio ca be localised by calculatig the differeces betwee the DC compoets of the sequece of frames i the header, startig from the first frame of the first blac sequece. The distace calculatio gives a sequece of differece images. Withi this sequece at least two pairs of differece images will show up a greater variace to the rest. I the first istace the differece betwee the last I- frame without text cotet ad the first I-frame with text cotet is relatively large, as is also the differece betwee the last I-frame with text cotet ad the followig I-frame without text cotet. The two frames with greater differeces ca be idetified directly [8]. After detectig the positio of the textual frames the respective frames ca be extracted from the MPEG compressed data strea coverted to the RGB colour model ad saved as a file i a stadard picture format, such as JPEG, GIF, TIFF, etc. The extracted frames are usually ot well suited for character recogitio as: the cotet frame is equivalet to a resolutio of 200 dpi or less. This is ot sufficiet for character recogitio, the text may be preseted i differet colours, the bacgroud may have various grey shades, which could be recogised as oise, the letters themselves ca cotai some oise ad bacgroud maes the separatio of text difficult. Applicatio of stadard OCR systems result i a uacceptable recogitio rate of partly less tha 10%. These problems ca be solved i part by image preprocessig. For optical character recogitio to perform well a moochrome image is eeded. Therefore the colour coded frame must be coverted first to a halftoe image. I the secod step the halftoe image is biarised usig a local adaptive threshold. Problems with oise ca be avoided by cosiderig eighbourig poits ad locally adjustig the threshold. Through the biarisatio uecessary grey levels i the bacgroud are elimiated. This simple method is ot adequate for recogisig text i the case of APTN textual frames. Parts of the bacgroud image ca oly be removed by direct search ad elimiatio of geometric shapes such as circles or filled boxes. I the case of other ews agecies similar adjustmets of the biarisatio process are eeded. Oce such a image processig chai has bee costructed, it ca be used for all textual frames from a particular ews agecy without sigificat chages. The text lie i the image ca be foud by projectio of the frame lies o the y-axis. Two text lies are separated by at least two bacgroud lies. Figure 2a shows a example of the described projectio. The small gaps i the projectio idicate the space betwee two text lies. The first ad the last pixel row of each text lie ca ow be determied. Short lies resultig from oise are elimiated by itroductio of a threshold value. The character segmetatio i a text lie is performed by projectig each text lie o the x-axis. Small gaps i the ru of the letter colour determie the place of the letter separatio. If this gap has a width of at least two bacgroud colums, the pixel group is idetified as a character. Due to the poor quality of the frames some letters do ot have sufficiet iterveig space. I the first ru they are thus cosidered as beig oe letter. The calculatio of the average letter width is the performed i a secod ru. If a sigle pixel group exceeds the average width by 50% the group must comprise two letters ad is separated. This method wors well whe fots with costat width are used. The character recogitio process is realised by a simple template matchig tool (Figure 2b). Firstly the broadcastig ews agecy must be determied. Subsequetly the appropriate set with the learig patter ca be loaded. The learig set is maually compiled. For compariso purposes a correlatio with a offset of two pixels i all directios is calculated ad added givig the sum describig the similarity of the examied patters. The assiged letter is the oe with the greatest correlatio sum. The extracted text is saved as a ASCII file. Figure 2: a) Example for the projectio o the y-axis; b) Character recogitio -4-
4. Cotet based search The extracted ey words are used as iput for a iteret search or for query machies of olie data bases. This additioal iformatio should assist ews editors i the compilatio of ewscasts. Such data icreases the umber of theme related ey words. These ca be used for improved retrieval of archived ews clips. Preset iteret search tools are ot suitable, as queries result i may sources beig searched givig huge, mostly useless sets of lis. A large percetage of iformatio retrieved is ot relevat or out of date. I some cases the page referred to does t exist ay more. Due to the geeral ey words available most of the other web pages have o direct relatio to the special topic desired. Followig all lis is time itesive ad cosequetly the editors' acceptace of the ew techology declies. A cocept for a ew, specialised search egie was thus developed. A automatic full text search is still executed but oly i a maually arraged set of reliable ad curret sources. Each site is visited i specified time itervals listed i a cofiguratio file. The updated iformatio is trasferred to the iteral server ad processed i order to remove all HTML ey words. All other iformatio is coverted ito ASCII form ad saved o the server. A advatage of this method is that miimal commuicatio with the iteret, from the view poit of the editor, is ecessary. Thus, a fast access to all relevat iformatio is give. Disadvatages result from judicial aspects. The public use of the ews archive is allowed. The savig of such data o the iteral server is ot. This is a major problem with regard to the archivig of the ewsfeeds. 5. ClaViPS: A presetatio system for ewsfeeds The methods described above form the basis for the ClaViPS-System which aalyses ad presets MPEG-1 compressed ewsfeeds over a computer etwor. With this system a TV ews editor ca quicly obtai a overview of recetly received ews material. This iformatio ca be combied with additioal related data from a database or a search of the iteret. The iformatio is preseted i the form of a HTML page o the editor s worstatio. The start page of this system gives a overview of the available video material, sorted accordig to the times whe the idividual items were received. The attached li refers to a idex page which lists the idividual ews items together with short cotet summaries. The editor ca select iterestig items from this list. Each item o the list has a separate WWW page assiged to it. This cotais title, locatio, time, laguage, ey words, extracted ey frames, additioal iformatio as well as a li to the MPEG compressed video clip. ClaViPS ca be visited at www.i.tu-clausthal.de/~mmedia/videosys. 6. Coclusios I this paper methods for adaptig ewscast productio worflow are preseted. Video aalysis algorithms are combied with optical character recogitio ad image processig methods i order to reliably extract iformatio from compressed ews feeds. These ca be used for specialised searches i ews archives. I order to give better support to ews editors i compilig ewscasts lis to relevat iformatio available o iteret are automatically geerated. The extracted ad compiled iformatio ca be stored i a way which will facilitate its future recovery. Future ivestigatios will be directed at geeralisig the video aalysis methods to eable their applicatio to MPEG-2 compressed video material. Furthermore, the used OCR methods must be adapted to the differet layouts of the ews agecies. The efficiecy of the specialised iteret search egie will be tested i a commercial televisio statio. Refereces [1] S. Geisler ad U. Luegjiraothai, Automatische Ereug vo Themewechsel i MPEG odierte Videos, Studiearbeit, TU Clausthal, 1998, 20 pages. [2] ISO/IEC 11172-1 (2,4), Iformatio techology --- Codig of movig pictures ad associated audio for digital storage media at up to about 1,5 Mbit/s ---, Part 1 (2,4): Systems, 1993. [3] B. Yeo, B. Liu, Rapid scee aalysis o compressed video, IEEE Trasactios o circuits ad systems for video techologie, vol.5, 1995, pp. 533-544. [4] G. Falemeier, Speicherplatzreduzierug ud Iformatiosaalyse vo MPEG-omprimierte Videos, Papierflieger Verlag, 1998. [5] R.M. Ford, C. Robso, D. Temple ad M. Gerlach, Metrics for scee chage detectio i digital video sequeces, IEEE Coferece o Multimedia Computig ad System 97, pp. 610-611, 1997. [6] I.K. Sethi, ad N. Patel, A statistical approach to scee chage detectio, Society of Photo-Optical Istrumetatio Egieers, 2420, pp. 329-338, 1995. [7] F. Irrgag, Schittereug i digitalisierte Filmsequeze, Master thesis, TU Clausthal, 1996. [8] G. Falemeier, G.R. Joubert ad O. Kao, Aalysis ad Processig of Compressed Newsfeeds, i J-Y. Roger: Techologies for the Iformatio Society: Developmets ad Opportuities, IOS Press 1998 [9] G. Falemeier, G.R. Joubert ad O. Kao, A system for Aalysis ad Presetatio of MPEG Compressed Newsfeeds, i J-Y. Roger,: Busiess ad Wor i the Iformatio Society, IOS Press, 1999 [10] K. She, E. Delp, A spatial-temporal parallel approach for real-time MPEG Video compressio, Image Compressio ad Graphics, 100-107, 1996. [11] K. She, E. Delp, A fast Algorithm for Video parsig usig MPEG compressed Sequeces, Proceedigs Iteratioal Coferece o Image Processig, IEEE Computer Society Press, pp. 252-255, 1995. -5-