Fast thumbnail generation for MPEG video by using a multiple-symbol lookup table

48 3, 376 March 29 Fast thumbnail generation for MPEG video by using a multiple-symbol lookup table Myounghoon Kim Hoonjae Lee Ja-Cheon Yoon Korea University Department of Electronics and Computer Engineering, 5-ka, Anam-Dong Sungbuk-Ku, Seoul 36-7 Korea Hyeokman Kim Kookmin University Department of Computer Science 86- Chungnung-Dong Songbuk-gu, Seoul 36-72 Korea Sanghoon Sull * Korea University Department of Electronics and Computer Engineering, 5-ka, Anam-Dong Sungbuk-Ku, Seoul 36-7 Korea E-mail: sull@mpeg.korea.ac.kr Abstract. A novel method using a multiple-symbol lookup table mlut is proposed to fast-skip the ac coefficients codewords not needed to construct a dc image from MPEG-/2 video streams, resulting in fast thumbnail generation. For MPEG-/2 video streams, thumbnail generation schemes usually extract dc images directly in a compressed domain where a dc image is constructed using a dc coefficient and a few ac coefficients from among the discrete cosine transform DCT coefficients. However, it is required that all codewords for DCT coefficients should be fully decoded whether they are needed or not in generating a dc image, since the bit length of a codeword coded with variable-length coding VLC cannot be determined until the previous VLC codeword has been decoded. Thus, a method using a mlut designed for fastskipping unnecessary DCT coefficients to construct a dc image is proposed, resulting in a significantly reduced number of table lookups LUT count for variable-length decoding of codewords. Experimental results show that the proposed method significantly improves the performance by reducing the LUT count by 5%. 29 Society of Photo-Optical Instrumentation Engineers. DOI:.7/.39978 Subject terms: Dc image; lookup table; MPEG-2; multiple-symbol lookup table. Paper 8437R received Jun. 4, 28; revised manuscript received Dec., 28; accepted for publication Jan. 2, 29; published online Mar. 26, 29. Introduction * Address all correspondence to Sanghoon Sull. 9-3286/29/$25. 29 SPIE Applications such as video browsing, storyboards, and scene change detection typically involve fast generation of reduced-size video frames or thumbnails. 3 Most thumbnail generation approaches for MPEG-/2 video streams generate dc images directly from a compressed video stream. A dc coefficient is a discrete cosine transform DCT coefficient for which the frequency is zero in both dimensions in a compressed block and is used to construct a dc image for a block encoded with a frame DCT. For a block encoded with a field DCT, a few ac coefficients are needed to generate a dc image, in addition to a dc coefficient. 4 6 However, since each of the ac coefficients in MPEG-/2 video streams is a codeword encoded with variable-length coding VLC, it is usually required that all ac coefficients or codewords should be fully decoded, since the bit length of a codeword coded with VLC cannot be determined until the previous VLC codeword has been decoded. Thus, for fast generation of a dc image from MPEG-/2 video streams, there is a need for an efficient scheme of decoding a dc coefficient and only the necessary ac coefficients without fully decoding all of them, especially in the case of digital HDTV devices, which usually have a lowpowered CPU. Several approaches have been reported to decode the data bit by bit bit-serial decoding. 7,8 In order to speed them up, instead of using bit-serial approaches, bit-parallel variable-length decoding approaches are proposed as more suitable for such video decoding applications. 9, The basic idea of these methods is to find the matched codeword with the stored symbols in a lookup table LUT so that one symbol is produced in a single cycle regardless of the length of the VLC codes. In addition to the bit-parallel variable-length decoding approaches, a method based on multiple-symbol parallel variable-length decoding has been proposed to detect multiple codewords and decode them in parallel in every clock cycle. However, those methods are still based on hardware for multisymbol decoding circuits. On the other hand, the proposed dc image generation method decodes only those codewords necessary for a dc image, which can be implemented in software on currently available consumer electronics platforms such as digital TVs or set-top boxes STBs. In this paper, a simple and efficient method using a multiplesymbol lookup table is proposed for decoding a dc coefficient and only those ac coefficients necessary to generate a dc image from MPEG-/2 I-frames, providing considerable reduction in processing time due to the reduced LUT count for variable-length decoding of codewords. Detailed descriptions are given in the following sections: Section 2 shows a fast DC image generation from I-frame, 376- March 29/Vol. 48 3

Sec. 3 presents the design of a multiple-symbol lookup table, and Sec. 4 provides the experimental results. Section 5 concludes the paper. 2 Fast DC Image Generation from I-Frames As shown in Fig., for a frame-coded macroblock X, where four 8 8 blocks X i, i 3 are encoded with a frame DCT, the generation of a dc image requires only four dc coefficients for each macroblock. However, for a fieldcoded macroblock X, where four 8 8 blocks X, i 3 are encoded with a field DCT, a dc image can be generated by fully decoding and reducing both top and bottom field blocks. To reduce the computation, a method using two additional ac coefficients dc+2ac was proposed in Ref. 4, and a method using one type of field blocks instead of using both top and bottom field blocks was introduced in Ref. 3. In this paper, to further speed up the generation of a dc image, for a field-coded macroblock, only one additional ac coefficient is utilized for each of either the top field blocks X,X or the bottom field blocks X 2,X 3. For a block X i, let R bea2 2 block R i, i 3, where R i isa reduced block obtained from the corresponding 8 8 spatial domain block P i of the block X i by reducing both horizontal and vertical resolutions by 8. Then, the reduced block R i, which is an average value for the 8 8 spatial block P i, can be written as R i = 64 V FP i H F = 64 V FC t X i CH F, where V F = ; H F =V F t, C is an 8-point DCT matrix, i.e., 2n + k C k,n = k cos, 6 = k 2 k =, otherwise, and i 3. Now, let R i bea2 reduced block obtained from the 8 8 spatial block P i of the top field block X i i by reducing the horizontal resolution by 8 and the vertical resolution by 4. Then, the reduced block R i can be written as R i = 32 V TP i H F = 32 V TC t X i CH F, where V T =, and H F and C are the same matrices as in Eq.. Let each DCT coefficient of a block X i be denoted as follows: X i for a dc coefficient, and X i ij for the ac coefficient at row i and column j in the block X i where i, j,. Then, from Eq. 2, when a macroblock is encoded with field DCT, the approximated reduced block R can be derived from a field-coded macroblock X by using only top field blocks as follows: 2 R i = R R, 3a R = 32 V TC t X CH F = 32 4. 3.625..273..85..72 X 4. 3.625..273..85..72 8. X +.96 X X.96 X, 3b R = 32 V TC t X CH F X +.96 X X.96 X, 3c 376-2 March 29/Vol. 48 3

X X X X X = X2 X3 frame-coded macroblock decoded macroblock (deinterlaced) X X X X X3 = field-coded macroblock 2 R Fig. Frame- and field-coded macroblocks and deinterlaced decoded macroblock. where only the first and second terms are used, since other terms, such as last six terms in R = X +.96 X +. X 2.38 X 3 +. X 4 +.26 X 5 +. X 6.8 X 7, are negligibly small due to their high-frequency characteristics. Figure 2 a and 2 b show the extraction of four dc coefficients representing four pixels for a macroblock in a dc image from a frame-coded macroblock and a field-coded macroblock, respectively, by Eq. 3. X2 X3 X X X X = X2 X3 ( X) ( X) R = ( X2) ( X3) (a) X R 3 Design of a Multiple-Symbol Lookup Table MPEG-2 VLC codewords are prefix-free codes having the property that no codeword may be the prefix of any other codeword, and therefore a codeword is uniquely decodable. From the property of unique decodability of the codeword, a concatenation of one or more codewords cannot be a prefix of any other multiple codewords. For example, Fig. 3 shows a binary code tree for the concatenation of two codewords represented by black leaf nodes. Using an original tree for single codewords represented by white nodes whose symbols are a, b, and c, a new tree is built simply by grafting a copy of the original tree onto each of its leaf nodes. The tree shows that each concatenation of two codewords whose symbols are aa,ab,...,cb, and cc has a different path from a root node to each leaf node, and therefore the concatenation of two codewords is also uniquely decodable. Thus, a multiple-symbol lookup table mlut can be built by which a uniquely decodable sequence of codewords can be fast-parsed simultaneously. DCT coefficient table Table B-5 in ISO/IEC 388-2 2 specifies the codewords ac coefficients of intra blocks with intra-vlc format, and it is observed that there are common prefix bits that provide the bit length of the codewords having the same bit length. For example, by just looking at the four starting bits that are the common prefix bits for the two VLC codewords s and s s is a sign bit, it can be found that the bit length of the codeword is 6, including its sign bit, whether the codeword is s or s. Since DCT coefficient table 2 indicates that the longest bit length of common prefix bits is 2, the minimum entry size of the mlut to cover all VLC codewords should be 496 2 2. Letting B be a bit sequence of a compressed MPEG-2 stream for a block A coded with VLC, the bit sequence B can be represented as X 2 X X X = X2 X3 X 3 ( X) +.96( X) ( X) +.96( X) R = ( X).96( X) ( X).96( X) (b) Fig. 2 Extracting four dc coefficients from a frame-coded and b field-coded macroblock. B = DC a a a 2...a n 2 EOB, 4 where DC denotes a codeword for the dc coefficient A, n is the number of ac coefficients, a j is a codeword for the jth ac coefficient j n, and EOB denotes the end-ofblock codeword. To construct a dc image, only one dc coefficient is required to be decoded when a block is coded with the frame DCT, whereas an additional ac coefficient A as in Eq. 3 is needed when a block is coded with the field DCT. The ac coefficient A can be obtained from a for alternate scan or a for zigzag scan according to the scanning order for DCT coefficients. After decoding the required codewords, the remaining codewords can be skipped by using the mlut whose entry value is the sum of the bit lengths of the concatenated code- a b c a a ab ac ba ca bb bc cb cc aa ab ac ba bb bc ca cb cc Fig. 3 Binary code tree for the concatenation of two codewords. 376-3 March 29/Vol. 48 3

Table An illustration of the proposed method: fast-skipping all unnecessary ac coefficients to construct a dc image from the example block A in Eq. 6. * The last bit in the buffer at lookup 7 belongs to the next block. LUT count no. 2-bit buffer S start index,end index 2-bit mlut lookup value: the bit length value is used to determine the next start index of S S, = =mlut 2 =mlut 3743 =bit length a +a : 2 S,22 = 2=mLUT 299 =bit length a 2 +a 3 +a 4 : 3 S 23,34 = 2=mLUT 2349 =bit length a 5 +a 6 +a 7 +a 8 : 4 S 35,46 = =mlut 689 =bit length a 9 +a : 5 S 45,56 = 2=mLUT 892 =bit length a +a 2 +a 3 : 6 S 57,68 = =mlut 273 =bit length a 4 +a 5 : 7 S 68,79 =* =mlut 289 =bit length a 6 +a 7 +EOB: words, i.e., multiple symbols. Consider a k-bit mlut in which the address of each entry is represented by k bits. Given a bit sequence B for a block, after performing variable-length decoding of the required codewords, the remaining bit sequence is first skipped by the entry value of mlut pointed to by the first k bits, and this process is repeated until the EOB is encountered. The ith entry for the k-bit mlut is calculated as follows: mlut i = l i j, n j= = n h if i h = EOB, 5 m otherwise, where m is the maximum number of codewords whose concatenated bit sequence matches the k-bit binary representation of the address i starting from the most significant bit, i j is the sequence of common-prefix bits of the jth codeword j m, and l i j is the bit length of the codeword having i j. Thus, the starting bit position of the next sequence i j+ after i j is determined by l i j. Note that a flag attached to each entry is used to indicate whether i m corresponds to EOB or not, to terminate the decoding of a block. For the example of a 2-bit mlut, the value of mlut 885 is according to Eq. 5, since the 2-bit representation of the address 885 contains two codewords and, whose lengths are 6 and 4. The value of mlut 34 is equal to 8, since the 2-bit representation of 34 has two codewords and EOB. As shown in Fig. 4, when a frame-coded macroblock is decoded whose VLC bit sequence S starts with, the first three bits corresponding to the value of DC is first processed, and then the next 8 bits can be skipped to the start bit position of the next block with LUT count 2 by utilizing the values of mlut 885 and mlut 34. The conventional LUT count would be 4 for the same bit sequence S to perform the variable-length decoding of three ac coefficient codewords and one EOB codeword. The whole process is illustrated using an example: Let an intra block A in the DCT domain be 5 3 3 2 3 2 2 A = 2 6. The dc coefficient can be coded with dc dct size and 376-4 March 29/Vol. 48 3

S: DC l(i ) l(i ) 2, 75, 5, Traditional LUT 2-bit mlut 4-bit mlut 6-bit mlut 8-bit mlut 2-bit mlut bits(mlut 885 ) first lookup l(i ) l(i :EOB) 8bits(mLUT 34 ) second lookup LUT count 25,, 75, Next block decoding 5, Fig. 4 An example of multiple-symbol lookup with 2-bit mlut. dct diff values such as and according to the MPEG-2 video coding standard. After zigzag scanning, the run, level sequence of ac coefficients will be,5,, 3,,,, 2,, 3,,,,,,,,, 2,,,2,,3,,2,,,,, 6,,,,,, EOB, and its corresponding bit stream sequence is a, a, a 2, a 3, a 4, a 5, a 6, a 7, a 8, a 9, a, a, a 2, a 3, a 4, a 5, a 6, a 7, EOB according to the DCT coefficient table Table B-5 in the MPEG-2 video coding standard. In the case of a frame-coded macroblock, we can skip all ac coefficients in the block. With the 2-bit mlut, from the ac coefficient bit stream S=, we can get the results in Table, showing only 7 LUT counts to parse 7 ac coefficients and the EOB code. In the case of a field-coded macroblock, we need first two ac coefficients a and a to build the dc image as described in Sec. 2. After parsing a and a,we can skip the 5 ac coefficients and EOB code in six LUT counts with 2-bit mlut. 25, 3 5 7 9 35792232527293333537 I-frame sequence number Fig. 6 Comparison of LUT counts required to perform the variablelength decoding for each of 38 I-frames of Table Tennis video sequence. 4 Implementation and Experimental Results The proposed algorithm was tested using k-bit mlut for two MPEG-2 streams: the Table Tennis video sequence 74 48, 8 Mbit/s and 4:2: format, and a real terrestrial HDTV broadcast stream 92 8, 9.4 Mbit/s, and 4:2: format. 9 The ratio of blocks (%) 8 7 6 5 4 3 2 Using traditional Using 2bit mlut Using 2bit mlut LUT 5 5 2 25 3 35 4 45 5 55 6 LUT count N Fig. 5 The ratio of the number of blocks requiring LUT count less than or equal to N to the total number of blocks during variablelength decoding of 38 I-frames in Table Tennis video sequence. Fig. 7 HDTV applications using the fast dc image generation method: a storyboard application for a music video program in HDTV, b browsing application, chaptering for an educational video program in HDTV. 376-5 March 29/Vol. 48 3

Table 2 Comparison of LUT counts per block and their reduction ratios in dc image generation using k-bit mlut. Count per block reduction ratio Video sequence Traditional LUT k-bit mlut k=2 k=4 k=6 k=8 k=2 Table Tennis 9.77 9.87 5.9% 8.64 56.28% 7.75 6.82% 7.2 64.49% 6.46 67.32% HDTV broadcast program 6.59 4.3 37.4% 3.77 42.77% 3.52 46.5% 3.32 49.62% 3.6 5.98% To evaluate the performance of the proposed algorithm, we compared k-bit mlut with the method using traditional LUT. Figure 5 shows the ratio of the number of blocks requiring LUT count less than or equal to N to the total number of blocks in order to extract dc images from 38 I-frames in the Table Tennis video. We can see that the proposed k-bit mlut scheme has larger fraction of blocks requiring small LUT count than does the traditional LUTbased one. It is also observed that the larger k yields the larger fraction of blocks. The proposed method can significantly reduce the required LUT counts per frame. For example, by using traditional LUT, the fraction of blocks requiring LUT count less than or equal to 2 is approximately 55%, whereas the fraction of parsed blocks becomes 95%and 99% with the 2-bit mlut and 2-bit mlut, respectively. Figure 6 shows that the required LUT count per frame can be significantly reduced by using the proposed method when 38 dc images are generated from I-frames in the Table Tennis video. From Table 2, it can be also seen that even the 2-bit mlut requiring only 4 kbyte of memory can reduce the LUT count per block by 5% for Table Tennis and 37.4% for the HDTV broadcast program. By increasing k, the LUT count can be further reduced at the expense of increased size of the mlut. The experimental results demonstrate that the proposed method using a k-bit mlut significantly reduces the LUT count compared with the methods using a traditional LUT for dc image generation from MPEG-/2 I-frames. Figure 7 shows an example graphical user interface of storyboard and browsing for HDTV programs using the fast dc image generation method, which is implemented on a commercial STB having a low-powered CPU. 5 Conclusion In this paper, a fast thumbnail generation method using the proposed multiple-symbol lookup table mlut has been presented, which could be used for video applications such as storyboard and video browsing for an HDTV digital video recorder DVR. Even though it needs an additional memory to construct the mlut, it can be applied to any device requiring real time or having a low-powered CPU, since it can perform fast variable-length decoding by skipping the unnecessary coefficients to construct a dc image. With the proposed scheme, we could improve the performance of dc image or thumbnail generation by reducing the LUT count by 5% with 2-bit mlut, requiring only 4 kbyte of table size, for the HDTV broadcast program, and it might be used for any application requiring variablelength decoding for fast skipping of unnecessary codewords. References. J.-C. Yoon, H. Kim, S. S. Chun, J.-R. Kim, and S. Sull, Real-time video indexing system for live digital broadcast TV programs, in CIVR 24: Int. Conf. on Image and Video Retrieval No. 3, pp. 26 269, Springer-Verlag 24. 2. B. Yeo and B. Liu, Rapid scene analysis on compressed video, IEEE Trans. Circuits Syst. Video Technol. 5 6, 533 54 995. 3. J.-R. Kim, S. Suh, and S. Sull, Fast scene change detection for personal video recorder, IEEE Trans. Consum. Electron. 49 3, 683 688 23. 4. J. Song and B. L. Yeo, Fast extraction of spatially reduced image sequences from MPEG-2 compressed video, IEEE Trans. Circuits Syst. Video Technol. 9 7, 4 999. 5. S. Suh, S. S. Chun, M.-H. Lee, and S. Sull, Efficient image down conversion for mixed field/frame-mode macroblocks, Electron. Lett. 39 6, 54 55 23. 6. J. Mukherjee and S. K. Mitra, Arbitrary resizing of images in DCT space, IEE Proc. Vision Image Signal Process, 52 2, 55 64 25. 7. A. Mukherjee, N. Ranganathan, and M. Bassiouni, Efficient VLSI designs for data transformation of tree-based codes, IEEE Trans. Circuits Syst. Video Technol. 38 3, 36 34 99. 8. Y. Ooi, A. Taniguchi, and S. Demura, A 62 Mbit/ s variable length decoding circuit using an adaptive tree search technique, in Proc. IEEE Custom Integrated Circuits Conf. CICC 94, pp. 7 994. 9. S. M. Lei and M. T. Sun, An entropy coding system for digital HDTV applications, IEEE Trans. Circuits Syst. Video Technol., 47 55 99.. S. W. Lee and I. C. Park, A low-power variable length decoder for MPEG-2 based on successive decoding of short codewords, IEEE Trans. Circuits Syst., II: Analog Digital Signal Process. 5 2, 73 82 23.. J. Nikara, S. Vassiliadis, J. Takala, and P. Liuha, Multiple-symbol parallel decoding for variable length codes, IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 2 7, 676 685 24. 2. ISO/IEC 388-2. Int. standard information technology Generic coding of moving pictures and associated audio information: Video, ISO/IEC JTC/SC29/WG 2. Myounghoon Kim received the BS and MS degrees in computer science from the Kookmin University, Seoul, Korea, in 23 and 25, respectively. He is currently working toward the PhD degree in electronics and computer engineering at Korea University. His research interests are video indexing for content-based retrieval, image processing, video signal processing, digital broadcasting, and other problems in image and video technologies. 376-6 March 29/Vol. 48 3

Hoonjae Lee received the BS degree in electronic engineering from Korea University, Seoul, Korea, in 26. He is currently working toward the MS and PhD joint degree in electrical engineering at Korea University. His research interests are image and video signal processing, digital broadcasting, and other problems in image and video technologies. object-oriented database systems. From 995 to 999, he was a member of the technical staff at the Multimedia Technology Research Laboratory, Korea Telecom, where he developed various video parsing algorithms and modeling schemes of video and their metadata. In 999, he joined the faculty of the School of Computer Science at Kookmin University, Seoul, Korea, where he is currently an assistant professor. His current research interests include video indexing and modeling, database, metadata, XML and XML applications for STB. Ja-Cheon Yoon received the BS and MS degrees in information engineering from the Sungkyunkwan University, Seoul, Korea, in 99 and 993, respectively, and the PhD degree in electronics engineering at Korea University, Seoul, Korea, in 24. From 993 to 999, he was with Korea Telecom Research Center, where he was involved in research and development on video conferencing systems. From 999 to 2, he was with KBS-Internet, where he was involved in design and development of an internet broadcasting system. He is currently working for VMAKR Korea, where he conducts research on digital video processing and its applications. His research interests are video signal processing and multimedia applications. Hyeokman Kim received his BS, MS, and PhD degrees in computer engineering from the Department of Computer Engineering, Seoul National University, Seoul, Korea, in 985, 987, and 996, respectively. He was with the Switching Technology Research Laboratory of Korea Telecom from 987 to 99, where he worked on developing a digital switching system. From 99 to 995, he was a research assistant at the Database Laboratory, Seoul National University, where he worked on data modeling, query processing, and Sanghoon Sull received the BS degree with honor in electronics engineering from the Seoul National University, Korea, in 98, the MS degree in electrical engineering from the Korea Advanced Institute of Science and Technology in 983, and the PhD degree in electrical and computer engineering from the University of Illinois, Urbana-Champaign, in 993. From 983 to 986, he was with the Korea Broadcasting Systems, working on the development of the Teletext system. From 994 to 996, he conducted research on motion analysis at the NASA Ames Research Center. In 996 and 997, he conducted research on video indexing and browsing and was involved in the development of the IBM DB2 Video Extender at the IBM Almaden Research Center. He joined the School of Electrical Engineering at Korea University as an assistant professor in 997 and is currently a professor. His current research interests include multimedia data management, including searching and browsing, image processing, internet applications, and digital broadcasting. 376-7 March 29/Vol. 48 3