MOBILVIDEO: A Framework for Self-Manipulating Video Streams

MOBILVIDEO: A Framework for Self-Maipulatig Video Streams Aath Grama, Wojciech Szpakowski, ad Vero Rego Departmet of Computer Scieces, Purdue Uiversity, W. Lafayette, IN 47907 fayg, spa, regog@cs.purdue.edu Λ ABSTRACT As hadheld computig devices ad cell phoes become commoplace, streamig media to these devices becomes a challegig problem. This is a result of severe badwidth, processig, ad memory costraits imposed by these devices. Badwidth limitatios ecessitate effective compressio strategies while memory ad processig costraits require iexpesive decompressio techiques. This eed for highly asymmetric compressio ad decompressio techiques is ot well addressed by covetioal stadards such as MPEG. I the absece of a uiversally accepted stadard, it is highly desirable to budle lea media hadlers with the media itself. Advaces i ifrastructure for mobile code (fast virtual machies, embedded device support) have eabled developmet of such active streams - media streams capable of self maipulatio. I this paper, we describe a highly asymmetric compressio/ decompressio techique, called 2D-PMC ad a associated mobile decompressor called MOBILVIDEO. We demostrate that the overhead of usig mobile code i our framework is miimal, the compressio comparable to MPEG2, ad decompressio possibleirealtimeat320 240 resolutio (the display resolutio of a ipaq 3870) ad 20 FPS eve o 206 MHz StrogARM class processors. 1. INTRODUCTION Dramatic growth i etworkig techology has fueled ew research, stadards, ad products for multimedia compressio ad decom- Λ This work was supported by NSF Grats CCR-9804760 ad CCR- 0208709, EIA-9806741, ACI-9875899, ad ACI-9872101, ad cotracts from sposors of CERIAS at Purdue. Computig equipmet used for this work is supported by the Itel Corp. pressio. As etworkig techologies evolve, ew challeges are beig posed for compressio techologies. Thi cliets relyig o wireless etworkig emphasize asymmetric compressio schemes that are ameable to extremely lea decompressio while yieldig good compressio ratios. Streamig media broadcasts over potetially cotested etworks require ovel ad powerful multimedia compressio schemes (e.g., [1, 4, 7, 8, 3, 13]). I [1] we proposed ovel multimedia compressio schemes based o approximate patter matchig that are lossy extesios of the well-kow Lempel-Ziv schemes. The cetral theme of a lossy extesio of Lempel-Ziv algorithm is the otio of approximate repetitiveess. If a portio of data recurs i a approximate sese, the subsequet occurreces ca be stored as direct or idirect refereces to the first occurrece. A highly desirable feature of our scheme is that the associated decompressor is extremely fast ad simple. This eables implemetatio i mobile code that ca be shipped alog with the media with little commuicatio overhead. Furthermore, eve o relatively outdated machies (Petium, 350MHz), our mobile code achieves real time performace at 25 frames/secod o frames of size 360 288 with high motio. We report here, the theoretical basis for our 2D-PMC compressio scheme alog with experimetal results from the MOBILVIDEO decoder. Our experimetal results idicate that the 2D-PMC decoder is a order of magitude faster tha a MPEG2 decoder ad similarly faster tha other streamig video decoders such as VIVOACTIVE,REALSYSTEM G2, ad JSTREAMING i ative form (machie code). This eables implemetatio of our decompressor i mobile code (bytecode) while achievig real-time performace. The implemetatio of both the 2D-PMC compressor ad MOBIL- VIDEO decoder are challegig problems from the algorithmic ad programmig stadpoits. Fidig a efficiet data structure for approximate search of multidimesioal sets i a huge multidimesioal database, is a iterestig problem i itself. We use a set of modified k-d trees ehaced by geeralized ru legth codig for approximate search. A key issue for high quality image ad video

compressio is the desig of a adaptive distortio measure that automatically adjusts its maximum distortio to produce perceptually high quality results. These fidigs were partially reported i [1]. For the MOBILVIDEO decoder, we have developed a highly optimized multithreaded implemetatio that is optimized for small footprit as well as real-time performace. 2. THEORETICAL UNDERPINNINGS We review ad exted here some theoretical results of [1]. Cosider a source sequece (X k ) takig values from a fiite alphabet A (e.g., jaj = 256), where k =(k 1;k 2;::: ;k d ) is a d-dimesioal idex. That is, X k : S!Awhere S is a two-dimesioal area (e.g., for images S is a N N square of pixels). To simplify the presetatio, we write Xm to deote cotiguous block of m+1 elemets (e.g., Xm = X r;s k=1;l=1 such that m +1 =r s). We formally should work with radom fields (e.g., Markov radom fields), however, we leave this precise formulatio of the problem to a exteded versio of the paper (the iterested reader is refereed to [8] for precise formulatio i terms of radom fields). Furthermore, we let P (x 1 ) deote the probability of X 1 = x 1.We ecode the source sequece X 1 ito a compressio code ad the decoder produces a estimate ^X 1 of X 1. More precisely, a code C is a mappig C : A!f0; 1g Λ, ad we write C (x1 ) for the compressio code of x 1, where lower-case letters represet realizatios of a stochastic process. Let `(C (x1 )) be the legth of a code (i bits) represetig x 1. The, the bit rate is defied as r(x 1 )=`(C (x1 ))= ad the average bit rate is defied as E[r(X 1 )] = E[`(C (X 1 ))]=: I passig we recall that the dimesio d is buried uder our otatio sice deotes the total volume i a d-dimesioal space; for example, if X would deote a sub-block of a d-dimesioal cube, the the above expressio would have bee E[r(X )] = E[`(C (X ))]= d. We oly cosider sigle-letter fidelity distortio measures d : A ^A!R + such that d(x 1 ; ^x 1 )= 1 X i=1 d(x i; ^x i): Our implemetatio of patter matchig video compressio is well modeled by the fixed database model [12]. I this model, the decoder ad the ecoder both have access to the commo database sequece ^X 1 (e.g., the first image i the video stream that could be viewed as a three-dimesioal data) geerated accordig to the distributio ^P. The source sequece X M 1 (e.g., M = N 2 for N N images) is partitioed accordig to Π ito variable legth phrases (i.e., rectagles) Z 1 ;Z 2 ;::: ;Z jπj of volumes L 1 ;::: ;L jπj, respectively. More precisely, for give fixed compressio boud D>0we defie L 1 = maxfk : d(x k 1 ; Z 1 = X L1 1 i+k 1 ^X i )» D; 1» i» k +1g; This implies that the first partitio comprises of the largest d dimesioal rectagle of the source X M 1 that matches a rectagle i the database ^X 1 to the specified tolerace. For example, for 2D data, the strig ^Z 1 recovered by the decoder is therefore give by: ^Z 1 = ^X i 1+k 1 1;i 2 +k 2 1 i 1 ;i 2 ; where k 1 k 2 = L 1. I a similar fashio (cf. [1]) we defie a sequece of subsequet partitios Π of volumes L m such that the source data X M 1 is parsed as X M 1 = Z 1 Z 2 :::Z jπj, while the decoder recovers ^Z 1 ^Z 2 ::: ^Z jπj, which is withi distortio D from X M 1. We represet each ^Z i by a poiter ptr to the database ad its legths. Therefore, its descriptio costs log + O(d log(l i )) bits. The total compressed code legth is X jπj M `(X 1 )= log + (dlog L i ); i=1 ad the bit rate (e.g., i bits per pixel) is give by X M r (X 1 )= 1 jπj log + (dlog L i ): (1) M i=1 To formulate our mai result, we itroduce the geeralized Shao etropy ^r 0(D) defied as: ^r 0(D) = lim!1 EP [ log ^P (B D(X1 ))] ; (2) where B D(x1 )=fy 1 : d(y 1 ;x 1 )» Dg is a d-dimesioal ball of radius D with ceter x 1,adEP deotes the expectatio with respect to P. Our mai theoretical result is as follows: THEOREM 1. Let us cosider the fixed database video model usig our 2D-PMC scheme with the database ^X 1 geerated by a Markovia source ^P, ad the source X M 1 emitted by a idepedet memoryless source P. The the average bit rate attais, asymptotically, the followig bouds ^r 0(D)» lim!1 lim M!1 E P ^P [r(xm 1 )]» 2^r 0(D): (3) Proof. Detailed proofs ca be foud i a exteded versio of the paper [1] focusig o 2D-PMC compressio. Remark. Our experimets idicate that the l.h.s. is ideed a accurate estimate of the bit rate. We should poit out that ^r 0(D) R(D) where R(D) is the optimal rate distortio fuctio. We observe, at least for memoryless sources, that ^r 0(D) ad R(D) do ot differ by much for moderate values of D. For a optimal patter matchig compressio the reader is referred to [7]. Armed with this theoretical uderstadig, we have developed a patter matchig compressio scheme that relies o approximate matchig to yield excellet compressio ratios.

3. REVIEW OF PMC VIDEO SCHEME I this sectio we review algorithmic ad implemetatio issues of the 2D-PMC video compressio scheme. 2D-PMC video compressio relies o a rage of techiques cetered aroud 2-D patter matchig used i cojuctio with variable adaptive distortio ad ehaced ru-legth ecodig. Two dimesioal patter matchig is the most efficiet ad computatioally expesive way of compressig images (frames) amog the methods used i 2D-PMC. The basic idea is to fid a twodimesioal regio (rectagle) i the ucompressed part of the image (e.g., the first frame i a group of pictures) that occurs approximately i the compressed part (i.e., database), ad to store a poiter to it alog with the width ad the legth of the repeated rectagle. Sice the objective is to search for the largest such area, a brute force search algorithm is too time cosumig. Cosequetly, we use k-d trees for acceleratig search. Ru-legth ecodig (RLE) of images idetifies regios of the image with costat pixel values. We ehace RLE by givig it the capability of codig regios i which pixel values ca be (approximately) modeled by a plaar fuctio. We call this techique ehaced ru-legth ecodig (ERLE). ERLE approximates i the least-squared error sese a give grid m of pixels by a plaar surface. The coefficiets of the plaar surface are computed by solvig a system of ormal equatios associated with the leastsquared error procedure. Oce the plaar surface is determied, a sub-segmet of the m grid that is withi the distortio distace is idetified ad coded usig ERLE. We observe that this is particularly useful for sythetic images typically foud o the Web. For lossless ecodig we use a custom-desiged arithmetic ecoder. However, to simplify ad speed up the decompressor, the arithmetic coder is disabled i the MOBILVIDEO decoder. It is our expectatio that as decoders computatioal capabilities improve, we ca eable arithmeticcodig i MOBILVIDEO without losig real-time performace. These three codig techiques i 2D-PMC (patter matchig, ehaced RLE, ad lossless codig) are applied to progressively code images. Assumig that a part of a image (frame) has bee previously ecoded, the key task is to ecode the pixels located to the right of ad below a poit we call the achor poit. The selectio of the ext achor poit is based o a growig heuristic (cf. [3]). The growig heuristic we adopt is the wave frot scheme. This scheme sweeps the image from top to bottom ad left to right. Other heuristics grow regios i a circular maer from a ceter of the image or expad from the mai diagoal. These heuristics have bee observed to yield similar results [3]. Oce the achor poit has bee idetified, the partial image is coded usig either 2-D patter matchig, ERLE, or lossless codig. We observe that for all our growig heuristics idividually coded subimages do ot overlap more tha twice. Our video compressio scheme uses a (decompressed) represetatio of the previous frame that has bee compressed i our framework as the database to compress curret frame. We refer to this decompressed represetatio as a lossy image. The reaso for usig the lossy image as a database is that we wat the decompressed image at the cliet side to be withi a costat distortio boud from the origial image. Sice the decompressor oly kows the compressed frame (i its compressed ad ucompressed form), this must be used as the database for patter matchig compressio. I cotrast, if the origial previous frame was used as the database, this database is ot available at the decompressor. Cosequetly, errors propagate quickly through subsequet frames durig decompressio. Sice the previous compressed frame (i its ucompressed form) forms a static database, the compressio process is modeled well by our aalytical framework of Sectio 2. 4. REAL-TIME DECOMPRESSION USING MOBILE CODE - MOBILVIDEO A key advatage of 2D-PMC is its highly asymmetric ature. While the computatioal cost associated with compressio is higher due to exhaustive search, the associated decompressio cost is much lower sice there are o floatig poit operatios. I typical video samples, we observe that compressio time is a order of magitude higher ad decompressio time is a order of magitude lower. This low decompressio time eables implemetatio of the decoder i mobile code while achievig real-time performace. This is i cotrast to covetioal schemes like MPEG that rely o frequecy domai trasforms defied over fixed size blocks. The associated decompressio schemes ted to be expesive ad their implemetatio o covetioal java virtual machies (JVMs) reders real-time performace difficult for meaigful frame sizes. A 2D-PMC compressed file is stored as a script. The three istructios correspodig to patter matchig, ehaced ru-legth ecodig, ad lossless codig are stored as two bit istructios followed by their argumets. These are evetually coded usig arithmetic codig. The etire decompressor method is show i the listig below. It is evidet that the method is extremely lea. I additio to the decompresspmc method, the oly other sigificat methods are readheader ad readbuffer, which read the header of the stream ad stream segmets i chuks of programmable size (we typically use 2KB segmets). A detailed demostratio of the mobile decompressor is available at http://www. cs.purdue.edu/homes/ayg/video/videos ad the decompressor code itself is available at http://www.cs.purdue.edu/homes/ayg/video/videos/player-0.99/. public void decompresspmc() { Date dstart = ew Date();

Date it it loop; ded; imi, isec; while (umops!= 0) { dstart = ew Date(); if (readi == 0 && byteoffset >= LEFT_BUFFER) { readbuffer(0, LEFT_BUFFER); readi = 1; graph.setcolor(color.white); graph.drawstrig("mm:ss " + imi + ":" + isec, 0, 120); isec = (it)(ded.gettime() - dstart.gettime()); graph.drawstrig("milli " + isec, 0, 140); switch (getbyteseg(size_of_op)) { case 0 : writepixel_bw(getbyteseg(size_of_lx)); break; case 1 : writerle_bw(getbyteseg(size_of_lx), getbyteseg(size_of_ly), getbyteseg(size_of_c0)-256, getbyteseg(size_of_cx)-256, getbyteseg(size_of_cy)-256); break; case 2 : writepatter_bw(getbyteseg(size_of_lx), getbyteseg(size_of_ly), getbyteseg(size_of_wid)-512, getbyteseg(size_of_hi) -512); break; umops--; if (pos >= MAX_POS) { drawimage(); ded = ew Date(); if ((ded.gettime() - dstart.gettime()) < 40) { loop = 4000; while(loop > 0) { loop--; ded = ew Date(); imi = ded.getmiutes()-dstart.getmiutes(); isec = ded.getsecods()-dstart.getsecods(); if (imi < 0) imi += 59; if (isec < 0) { imi++; isec += 59; Figure 1: Screeshot of the MOBILVIDEO decompressor illustratig video without plugis with excellet compressio ad decompressor performace. The MOBILVIDEO decompressor for 2D-PMC is fashioed as a Java applet that commuicates the image or video segmet, decompresses it, ad reders it. To optimize the performace of MOBILVIDEO, its fuctioality is implemeted i three threads - reader, decompressor,adreder. These threads commuicate via sychroized (mutually exclusive) operatios o shared buffers. The reader ad decompressor threads commuicate via the iput (i) buffer ad decompressor ad reder threads commuicate via the display buffer. Compressed data is copied ito the iput buffer i (programmable) chuks of size 2KB. The size of this chuk is selected to optimize etwork latecy, buffer maagemet overheads, ad compressed image sizes. Decompressed data is copied ito the display buffer oe frame at a time. A screeshot of the MOBILVIDEO decompressor is illustrated i Figure 1. The reader thread reads a parcel of data ad places it i the iput buffer if space is available, otherwise it yields (coditio waits). The decompressor reads i data from the iput buffer as it becomes available. If there is o data i the iput buffer, it yields as well. If there is data, it attempts to decompress the data ad place it ito the display buffer whe a etire frame has bee decompressed. If the display buffer is full, the decompress thread yields as well. The reder thread reads i frames from the display buffer as they become

available ad reders them at programmed itervals. Threads are programmed to maximize cocurrecy ad miimize sychroizatio overheads. Buffer sizes are selected to optimize serializatio overheads as well as for latecy tolerace. 5. EXPERIMENTAL RESULTS I this sectio, we preset detailed experimetal results correspodig to compressio ratios, compressio time, ad decompressio time. The objective of this exercise is to demostrate that 2D-PMC achieves compressio rates similar to MPEG2 while supportig decompressio rates roughly a order of magitude higher. The correspodig compressio rates are roughly a order of magitude lower for 2D-PMC. We demostrate these i the cotext of a variety of video segmets. Our experimetal demostratios are based o five video segmets, each with distict characteristics. Samples Claire ad Missa correspod to ews broadcasts. The backgroud is static ad the motio is limited. Cosequetly, most compressio techiques yield excellet compressio for these video segmets. Segmets Football, PomPom, ad Pig-Pog (Figure 2) have sigificat motio i them ad some scee chages. Segmet Trai (Figure 3) correspods to a cartoo clip. This segmet is selected to demostrate the superior characteristics of 2D-PMC for sythetic video segmets. 5.1 Compressio Rates for MPEG2 ad 2D- PMC Our first experimet compared the compressio rates achieved from MPEG2 ad 2D-PMC. We selected a group of pictures to correspod to te frames i each case. Data rates were computed from the sizes of each group of pictures. The parameters i 2D- PMC were selected to esure that video quality is comparable to or better tha that of MPEG2. For a full demostratio, we direct the reader to http://www.cs.purdue.edu/homes/ayg/ Video/. We observe from Table 1 that the performace of MPEG2 ad 2D-PMC is comparable for a wide rage of segmets with variatio from 7.9% worse (i case of Claire) to 31.3% better (i case of sythetic video segmet Trai). 2D-PMC has a umber of programmable parameters such as search regio, umber of seed poits, umber of templates, etc., that allow tradeoffs betwee compressio time, video quality, ad compressio ratios. The results preseted i Table 1 are selected as the best compromise betwee image quality ad compressio time. If a better compressio ratio is desired, it is easy to chage the parameters of the compressio script to explore better matches at the expese of icreased compressio time. Figure 2: Sample video Pig Pog origial (left) ad PMC compressed (right) for illustratio.

Sample MPG PMC Comp. Time Decomp. Time MPG PMC MPG PMC Claire 17.7 19.1 2 26 0.36 0.05 Football 111.9 90.9 3 29 0.34 0.09 Missa 20.4 20.2 9 23 0.32 0.03 PomPom 187.1 174.6 7 34 0.35 0.07 PigPog 113.8 104.9 8 39 0.35 0.03 Trai 202.8 139.3 9 25 0.35 0.04 Table 1: Compariso of data rates (KB/s), compressio, ad decompresio times (i secods o a Petium, 350MHz, 128MB) from five differet samples illustratig that 2DPMC yields performace ragig from 7.9% worse to 31.3% better tha MPEG2. The sizes of samples Claire, Football, Missa, Pom- Pom, Pig Pog, ad Trai are 360 288, 360 243, 360 288, 352 288, 360 256, ad 352 288 pixels respectively. The platform is itetioally chose to be a computatioally weak oe (the weakest we could fid i our lab) to demostrate the highly assymmetric ature of 2D-PMC. 5.2 Compressio Times for MPEG2 ad 2D- PMC The compressio times for MPEG2 ad 2D-PMC are preseted i Table 1. The compressors are implemeted i ative code as opposed to mobile code. This is motivated by the assumptio that servers are likely to have sigificat computatioal resources. We observe that 2D-PMC is 3- to 10-times slower tha MPEG2 i terms of compressio time. The best performace for 2D-PMC is for sythetic segmets (e.g., Trai). The sigificat time premium of 2D-PMC ca be attributed to its more exhaustive search. This search time is a sesitive fuctio of various parameters - the umber of seed templates, search regio for seed poits, ad umber of seed poits. I geeral, sigificat improvemets i compressio times ca be achieved with miimal degradatio i compressio ratios. Figure 3: Sample video Trai origial (left) ad PMC compressed (right) for illustratio. 5.3 Decompressio Rates for MPEG2 ad 2D- PMC Decompressio times represet the most sigificat advatage of 2D-PMC over existig compressio stadards. Sice decompressio i 2D-PMC is simply a set of poiter lookups or icremet operatios i case of ERLE, it is extremely fast. I Table 1, we compare the decompressio times for MPEG2 ad 2D-PMC. The times correspod to total decompressio time for a group of te pictures usig ative (machie) code istead of mobile code. This is because a mobile decompressor for MPEG2 was uavailable for compariso, therefore, for a fair compariso, ative code was used i both cases. It is evidet from the table that MPEG2 decompressio takes about 34ms/frame o a average whereas 2D-PMC takes

roughly 4 ms/frame. This implies that our mobile implemetatio of a 2D-PMC decompressor easily achieves real-time performace eve o older platforms such as a Petium 266/Widows platform. 5.4 Discussio of Performace ad Related Systems There have bee other efforts at developig mobile decompressors for a variety of compressio techiques. The closest i terms of project objectives is the JSTREAMING H263 decoder from the Multimedia Commuicatios Research Lab at the Uiversity of Ottawa (http://www2.mcrlab.uottawa.ca/ jauvae/ H263Decoder/JDK1.1/). Beig a H263 decoder, it is suited for low bit-rate low-motio video. Similar products are also available from EMBLAZE(http://www.emblaze.com). There have also bee efforts at mobile MPEG2 decoders. However, the performace of these decoders is far from real-time for higher quality video. [13] E.H. Yag, ad J. Kieffer, O the Performace of Data Compressio Algorithms Based upo Strig Matchig, IEEE Tras. Iformatio Theory, 44, 47-65, 1998. [14] Z. Zhag ad V. Wei, A O-Lie Uiversal Lossy Data Compressio Algorithm via Cotiuous Codebook Refiemet Part I: Basic Results, IEEE Tras. Iformatio Theory, 42, 803-821, 1996. 6. REFERENCES [1] M. Alzia, W. Szpakowski ad A. Grama, 2D-Patter Matchig Image ad Video Compressio, IEEE Tras. o Image Processig, 11, 318-331, 2002. [2] G. Cokli, G. Greebaum, K. Lilevold, A. Lippma, ad Y. Rezik, Video Codig for Streamig Media Delivery o the Iteret, IEEE Trasactios o Circuits ad Systems for Video Techology, 2001, to appear. [3] C. Costatiescu ad J. A. Storer, Improved Techiques for Sigle-Pass Adaptive Vector Quatizatio, Proc. IEEE, 82, 933-939, 1994 [4] W. Fiamore, M. Carvalho, ad J. Kieffer, Lossy Compressio with the Lempel-Ziv Algorithm, 11th Brasilia Telecommuicatio Coferece, 141-146, 1993. [5] J. Gibso, T. Berger, T. Lookabaugh, R. Baker Multimedia Compressio: Applicatios & Stadards, Morga Kaufma Publishers 1998. [6] JSTREAMING - H263 Video Decoder 1.8. [7] I. Kotoyiais, A Implemetable Lossy Versio of the Lempel-Ziv Algorithm Part I: Optimality for Memoryless Sources, IEEE Tras. Iformatio Theory, 45, 2285-2292, 1999. [8] I. Kotoyiais, Patter Matchig ad Lossy Data Compressio o Radom Fields, preprit 2002. [9] T. Łuczak ad W. Szpakowski, A Suboptimal Lossy Data Compressio Based i Approximate Patter Matchig, IEEE Tras. Iformatio Theory, 43, 1439 1451, 1997. [10] W. Szpakowski, Average Case Aalysis of Algorithms o Sequeces, Joh Wiley & Sos, New York, 2001. [11] Vivo Software, VIVOACTIVE software documetatio, http:// www.vivo.com. [12] A.J. Wyer, The Redudacy ad Distributio of the Phrase Legths of the Fixed-Database Lempel-Ziv Algorithm, IEEE Tras. Iformatio Theory, 43, 1439 1465, 1997.