A Real-Time Encoding and Decoding System for Nonlinear HDTV Editor

Transcription

1 A Real-Time Encoding and Decoding System for Nonlinear HDTV Editor Chul Soo Lee, 1 JoonHong Park, 1 DooSoo Yoon, 1 JaeHo Jeon, 1 Hyun Wook Park, 1 Ji Hee Yeo, 2 Jong Hwa Lee 2 1 Department of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Kusong-dong, Yusong-gu, Taejon, Korea. hwpark@athena.kaist.ac.kr 2 KBS Technical Research Institute, Seoul, Korea ABSTRACT: A real-time encoding and decoding system (REDS) for HDTV that can be used for nonlinear HDTV editing in studio has been developed. The intrapicture coding of motion JPEG is implemented and optimized in the REDS so that a high-quality image can be recovered for the nonlinear editing. The REDS has a parallel architecture with multiple programmable digital signal processors (DSP) and reconfigurable field programmable logic devices (FPLD). The HDTV image is spatially partitioned and concurrently processed by the multiple processors. The programmable DSPs perform the discrete cosine transform and quantization to reduce the spatial redundancy of the HDTV image, whereas the FPLDs perform the variable length coding to reduce the statistical redundancy. In addition, field-based quantization matrices are developed for HDTV images. The REDS has the programmability and the random accessibility of image frames, the two most important features for a nonlinear HDTV editing system John Wiley & Sons, Inc. Int J Imaging Syst Technol, 11, , 2000 I. INTRODUCTION Multimedia data processing is now expected to be the driving force in the evolution of computing, communication, and broadcasting technology. Recently, digital technology has been applied to TV broadcasting (Bhatt and Birks, 1997). Digital image processing is one of the most powerful means to extend the capability of television services in both studio and broadcasting systems. As the resolution of digital TV increases, data amount and computational complexity also increase. If the HDTV image is stored without compression, memory space of about 7 Gbytes is required for every minute. In order to reduce the amount of data, several multimedia video processors (MVPs) have been developed for the HDTV video decoder (Lee et al., 1996) that comply with MPEG-2 MP@HL. However, it is not easy to implement the realtime encoding and decoding of HDTV images with an available single-chip processor. In order to edit and generate video sequences in a studio, a nonlinear editing system, which can edit one or multiple video DooSoo Yoon is now with Digital Media Lab., LG Electronics Inc., Korea. JaeHo Jeon is now with Telecommunication Research and Development Center, Information and Communication Business, Samsung Electronics Co., Korea. Correspondence to: Hyun Wook Park. Grant sponsor: Korean Broadcasting System (KBS). streams in a frame-by-frame fashion, is required. The biggest difference between nonlinear and linear editing is whether users can access the data randomly or not. The random access of a frame is an essential feature for nonlinear editing. Therefore, intrapicture coding is more convenient for nonlinear editing rather than interpicture coding of MPEG (ISO/IEC, 1995). In addition, the image details should be preserved in the studio, where the image quality is much better than that of the broadcasting image. We have developed a real-time encoding and decoding system (REDS) for nonlinear HDTV editing in the studio. The REDS is equipped with a TMS320C80 (Texas Instruments [TI], 1995a) MVP as a processing element (PE). The MVP is a heterogeneous multipleinstruction multiple-data (MIMD) multiprocessor. The REDS is a fully programmable system that allows a compression algorithm of any kind to be programmed in the system. An intrapicture coding algorithm of a motion JPEG (Wallace, 1991) has been implemented and optimized for a high image quality and a real-time operation in the REDS. This paper presents system requirements and architecture of the REDS in Section II. Section III describes field-based quantization matrices for HDTV images. Further optimizations of the system are described in Section IV, followed by the conclusion in Section V. II. ARCHITECTURE OF THE REDS A. System Requirements of the REDS. The REDS is a realtime encoding and decoding system for HDTV application. Several requirements for the system and the solution of each requirement are described as follows: Real-time processing: Real-time processing is required for playing and recording HDTV video sequences. Considering the amount of compression, it should have a parallel architecture with multiprocessors. Programmability: Because there exist many different algorithms for image compression such as JPEG, MPEG, EZW (Shapiro, 1993), and SPIHT (Said and Pearlman, 1996), it should be possible for the various algorithms to be implemented in the system without changing the system hardware architecture. Thus, we adopt a general-purpose programmable digital signal processor (DSP) as a PE John Wiley & Sons, Inc.

2 Figure 1. Block diagram of the REDS for nonlinear HDTV editing. Fast data access: Because the system manipulates a huge amount of data, the data transfer bandwidth is one of the most important factors influencing system performance. We have designed two independent data buses, VBUS and HBUS, to interchange the uncompressed and compressed data between the functional units, respectively. High-quality image: Because the system is designed to be used in a studio, the image quality should be better than that of a broadcasting image. The intrapicture coding algorithm of the motion JPEG is used and the quantization matrix is designed for field-based HDTV images. Auxiliary processor for bit-wise operation: Although the MVP has a high computational power, it is not efficient for the MVP to carry out the bit-wise operations such as Huffman coding and decoding. Therefore, a field programmable logic device (FPLD) is used for Huffman coding and decoding. B. System Overview. Figure 1 shows the overall block diagram of the REDS. The REDS is made up of three functional units: the video unit (VU), the multiple processing units (PUs), and the host unit (HU). The VU provides the SMPTE-260M interface (SMPTE, 1995) for the HDTV digitized video and performs the spatial partitioning of field images for multiple PUs. The PUs execute the compression and decompression processes of the spatially partitioned data in parallel. The HU provides an interface with the parallel disk system to store the compressed data. VBUS and HBUS are implemented to exchange the uncompressed data between the VU and the PUs and between the PUs and the HU, respectively. The bus width and the operation frequency are determined to meet the real-time processing requirements on the large amount of data to be transferred. C. VU. A block diagram of the VU is shown in Figure 2. The VU performs the video input/output (I/O) interface that complies with Figure 3. Dimension of a field, a strip, and a block of HDTV image. the SMPTE-260M (SMPTE, 1995), partitioning and distributing the video data to the PUs in the encoding procedure and combining the decompressed video data from the PUs in the decoding procedure. Based on the SMPTE-260M, the input video data rate is MHz with two 8-bits. Because the rate is too fast to handle the data directly, we reduced the clock rate using time multiplexing and bus-matching FIFOs to about 9.3 MHz with two 64-bits. The image data from the SMPTE-260M interface are stored in the picture memory. The data amount of a field image is (E Y and E Cb/Cr )inay:c b :C r 4:2:2 format. The field image is partitioned into strips, each of which consists of pixel blocks as shown in Figure 3. For easy partitioning of the field data, the dimension of the field image is changed to 2, , with 128 redundant vertical lines and 28 uncovered horizontal lines. Then, the image data in each strip are transferred to each PU through the VBUS. The data transfer rate of the VBUS is 20 MHz with 64 bits. It is sufficient to transfer all video data through the VBUS with some data guard time. In the decoding procedure, the decompressed data move from the PUs to the picture memory of the VU through the VBUS, and are transferred to the SMPTE-260M interface. The picture memory is composed of dual-bank field memories for uncompressed field data where each bank has three ports for the SMPTE-260M, VBUS, and MVP interfaces. The MVP in the VU has two roles: to observe whether the VU operates properly or not and to perform simple video editing such as rotating, inverting, and warping. The FPLDs control this arbitration according to the operation mode, which is determined by the MVP in the HU. When one picture memory is used for the SMPTE-260M interface, the other picture memory is used for interface with the PUs through the VBUS. The command queue is implemented with a dual-port memory for receiving/transmitting the command and some parameters from/to the HU. Figure 2. Block diagram of the VU. D. PU. The function of the PU is to perform the intrapicture coding, i.e., the discrete cosine transform (DCT) and quantization to reduce the spatial redundancy, and the variable length coding to reduce the statistical redundancy. The PU is composed of the programmable digital signal processor (DSP), FPLDs, and several kinds of memories (Fig. 4). Vol. 11, (2000) 153

3 Figure 4. Block diagram of the PU. We adopt the general-purpose DSP as a PE to achieve the programmability. The DCT is used for transform coding, which can be implemented by using a DCT-dedicated application-specific integrated circuit (ASIC). However, it may limit any improvement or change of compression algorithms. In adopting the general-purpose DSP, it can be easy to upgrade the system for further improved algorithms. If a new and more powerful algorithm is developed, it can be implemented by modifying the DSP programs without any change of hardware architecture. We used the TMS320C80 (called MVP; TI, 1995) as a general-purpose DSP. The MVP consists of four advanced DSPs (ADSPs), a master processor (MP), a shared memory, and a transfer controller (TC), which operate in parallel. We have implemented the inverse DCT (IDCT) of an 8 8-pixel block, which requires 617 cycles of an ADSP in the MVP operating at 40 MHz (Jeon et al., 1998). The total number of blocks to be transformed by the IDCT are 1,966,080 per second ( [pixels per frame] 30 [frames per second]/64 [pixels per block]). Therefore, eight MVPs are required for the real-time operation of the 8 8-block IDCT as follows: 1,966,080 (blocks/sec) 617 (cycles/block) 25 (nsec/cycle) 4 (ADSPs/MVP) 7.6 (MVPs) The ADSPs and MP of the MVP can perform various image and video processing algorithms in parallel. However, they cannot provide an efficient processing power for a bit-wise operation such as Huffman coding and decoding. Therefore, the MVP performs the DCT and IDCT, whereas Huffman coding/decoding (i.e., variable length coding [VLC]/decoding [VLD]) is implemented in the FPLD, which is described in Section IVB. E. HU. Functions of the HU are to control the VU and the PUs and to transfer the compressed data from the REDS to the parallel disk system in the encoding procedure, and from the parallel disk system (Park et al., 1998) to the REDS in the decoding procedure. The encoding and decoding operations are synchronized with the vertical sync signal, which is extracted from the SMPTE-260M interface in the VU. Figure 5 shows the block diagram of the HU. Data transfer through the HBUS between the HU and PUs is arbitrated by the HU, i.e., eight PUs can transfer the compressed data to/from the HU sequentially according to control of the HU. Because the amount of compressed data is much smaller than the original image data amount, we have designed the bus width of the HBUS to be 32 bits and the VBUS to be 64 bits. In the encoding procedure, the compressed data are transferred from the PUs to the HU according to the sequence of the PUs, i.e., the HU waits for the compressed data until the selected PU loads the compressed data on the HBUS in order. In the decoding procedure, the HU divides the time interval uniformly to distribute the compressed data sequentially to the PUs. Because the compressed data are small compared with the bus throughput, they can be transferred to the PUs with sufficient time intervals. Like the VU, the HU has dual-bank field memories, where each bank can be accessed independently. When one bank is used to interchange the compressed data between the HU and the PUs, the other is used for data transfer between the HU and the parallel disk system. Even though the compressed data amount of a field is different from one to the other, the parallel disk system should store a fixed size data for a field to randomly access the compressed data in the disk system. In order to keep the compressed data amount constant, the HU appends the redundant data after the compressed data. Then, the predefined amount of the compressed data for each field is transferred to the parallel disk system. The SCSI-II interface (Fig. 5) is implemented for interfacing with a host computer. The program and parameters of the compression algorithm can be loaded from the host computer to the REDS through the SCSI interface. III. FIELD-BASED QUANTIZATION MATRICES When the DCT coefficients are quantized, the scaled quantization matrix is normally used to adjust the image quality and to control the bit budget. In REDS, a new quantization matrix is developed to achieve better quality of the field-based HDTV images. We used the rate-distortion optimized (RD-OPT) algorithm, proposed by Ratnakar and Liviny (1995), for constructing the quantization matrix with an optimal rate distortion. The algorithm analyzes the distribution of DCT coefficients over the wide range of bit rates and distortions, and then an optimal quantization matrix can be obtained by dynamic programming. However, the optimal quantization matrix obtained from a specific image may not be optimal for the other images because the quantization matrix is optimized just for the specific image. If the quantization matrix were optimized for each image, it would be impossible to meet the real-time processing with the optimal quantization matrix because the optimization process for each image requires a large amount of computation. Therefore, a quantization matrix of greater robustness is required for field-based HDTV images, which can be applied for all HDTV images and just scaled by Q factors for various bit-rate budgets. We obtained the robust quantization matrices by averaging the R-D optimal quantization matrices that are obtained from many experiments with 31 HDTV images. The developed field-based and JPEG quantization matrices for luminance and color components are shown in Tables I IV. The components of the developed quantization matrices are more antisymmetric to the diagonal than the JPEG quantization matrices. There are objective and subjective measures in evaluating the image quality. The subjective evaluation is not very reliable nor Figure 5. Block diagram of the HU. 154 Vol. 11, (2000)

4 Table I. Developed quantization matrices for field-based HDTV images of the luminance component Table II. Developed quantitation matrices for field-based HDTV images of chrominance compartments reproducible. Therefore, the objective measure of PSNR is used to evaluate the decompressed images in this paper, even though it does not fully take into account the properties of human perception. Figure 6 shows the PSNR comparison for the different quantization matrices in the luminance component of HDTV field images. The simulation result shows that the reconstructed image using the field-based quantization matrix has a better PSNR than the JPEG quantization matrix. Almost the same results were obtained for the chrominance components, even though the amount of improvement is slightly less than that of the luminance component. IV. FURTHER OPTIMIZATIONS IN THE PU The motion JPEG algorithm is used to compress the HDTV images for studio editing. The motion JPEG algorithm consists of the 8 8-block DCT to reduce the spatial redundancy and the VLC to reduce the statistical redundancy. In order to utilize the processing resources efficiently, the block DCT and quantization processes are performed by the MVPs, whereas the VLC is performed by FPLDs. This section describes the analysis of the execution and data transfer times in the MVP and the FPLD. A. Analysis of the Execution Time. One field of HDTV image consists of strips, each of which has pixel blocks as shown in Figure 3. Because the REDS is equipped with eight MVPs, one MVP should encode or decode 128 strips in 1/60 s to realize the real-time processing, i.e., the processing time per each strip should be less than 130 s. The MVP consists of five on-chip processors of one MP and four ADSPs, and a TC. The TC provides fast DMA transfer from any external memory to on-chip memory and vice versa (TI, 1995b). Four ADSPs perform the block DCT or IDCT, and the MP controls the jobs of the ADSPs and the TC. We divide the internal memory of the MVP in the PU into two parts. First, the TC loads the data for the DCT or IDCT from the local memory to one part of the internal memory. After finishing the data transfer, the TC waits for the command from the MP. One MP orders the TC to restart the data transfer and orders ADSPs to start Table III. JPEG quantization matrices of the luminance component Table IV. JPEG quantization matrices of chrominance components the DCT or IDCT; the TC performs data transfer to the other part of the internal memory. That is, the ADSPs perform DCT/IDCT of the data in one part of the internal memory, whereas the TC performs the data transfer to/from local memory with the other part of the internal memory. Therefore, ADSPs can perform the DCT or IDCT without any delay for data transfer. The execution time in the ADSP and the data transfer time in the TC are analyzed as follows: The IDCT process in the decoding procedure requires the longest execution time among several processes of motion JPEG encoding and decoding (Jeon et al., 1998). Thus, the execution time of the IDCT of an 8 8 block is analyzed in this paper. The required number of cycles for IDCT is 617 cycles in an ADSP. Thus, the execution time of an MVP for IDCT of a half strip is given as follows: 16 (blocks/half strip) 617 (cycles/block) 25 (ns/cycle)/4(adsps/mvp) 61.7 s/half strip A strip of the encoded data from the HU is variable and run length decoded by the decoding FPLD. The decoding FPLD requests the MVP on the PU to take a half strip of the run-length decoded data. While a half strip is processed in the ADSPs, the other half strip is stored into one part of the internal memory of the ADSPs. Figure 7 shows the data flow for the MVP to perform the IDCT and to transfer the decoded data to the VU. The data transfer time from the decoding FPLD to the internal memory of the MVP [(1) of Fig. 7] is given as: 8 8 (pixels/block) 2 (bytes/pixel) 16 (blocks/half strip) 50 (ns)/4 (bytes) 25.6 s/half strip where the clock duration is 50 ns, the data bus is 32 bits wide, and the encoded data are 16 bits wide. The processed data of a half strip is stored into the local memory from the MVP [(2) of Fig. 7], where its transfer time is given as: 8 8 (bytes/block) 16 (blocks/half strip) 25 (ns/cycle)/8 (bytes/cycle) s/half strip Vol. 11, (2000) 155

5 Figure 6. PSNR comparison for the JPEG and the proposed quantization matrices in the luminance component of HD850 image. The clock duration for the local memory is 25 ns and the data bus is 64 bits wide. The result of the IDCT is the decompressed 8-bit image data. To make a suitable format for the VBUS interface, eight accesses to internal memory are required for word-length conversion from 8 to 64 bits. After the half strip is decompressed and stored in the local memory, the decompressed data are transferred to the VU through the VBUS interface. For this data transfer, the decompressed data are loaded into the MVP [(3) of Fig. 7], where it requires the data transfer time as: 8 8 (bytes/block) 16 (blocks/half strip) 25 (ns/cycle)/8 (bytes/cycle) 3.2 s/half strip The decompressed data in the MVP are written into the VBUS interface [(4) of Fig. 7] to be transferred to the VU with the transfer time as: 8 8 (bytes/block) 16 (blocks/half strip) 50 (ns/cycle)/8 (bytes/cycle) 6.4 s/half strip Figure 7. The data flow for MVP to perform the IDCT in decoding procedure. The clock duration for the VBUS interface is 50 ns and the data bus is 64 bits wide. Figure 8. Execution and data transfer times for IDCT of a strip in MVP. Numbers of (1), (2), (3), and (4) indicate the corresponding data transfer in Figure Vol. 11, (2000)

6 The time analysis in Fig. 8 is proved in the actual system. The total times of the data I/O and the processing in the MVP are and s for a half strip, respectively, which are less than 130 s. Therefore, the REDS can perform the real-time processing of HDTV decoding. In the conventional algorithms of moving picture coding such as MPEG, the encoder contains the whole decoder components. Therefore, the encoding time is usually longer than the decoding time. However, the REDS is designed on the base of motion-jpeg and the job assignment of encoding is very similar to that of decoding. In addition, the DCT is slightly faster than the IDCT in the MVP. Therefore, the encoding time is slightly faster than the decoding time. B. VLC and Variable Length Decoding (VLD). A VLC of the Huffman coding is applied to reduce the statistical redundancy after the DCT and quantization. Although the MVP is a very powerful processor, it is not efficient to process simple bit manipulations. Thus, we implement the Huffman coding using the FPLD. The operating frequency of the encoding FPLD is determined by considering the real-time processing requirement. Each strip consists of bit DCT coefficients, which should be encoded by the encoding FPLD within 130 s. To produce a Huffman-encoded DCT coefficient in each cycle, the cycle time should be less than 63.4 ns for a real-time processing. Therefore, the operation frequency of the encoding FPLD is determined to be 20 MHz. After the DCT and quantization, the MVP writes the quantized DCT coefficients into the encoding FPLD. The encoding FPLD consists of two parts, the run-length coder (RLC) and the VLC. First, the RLC counts the number of zero coefficients. Once an input coefficient is a nonzero value, the RLC generates a run-level pair of zero-run and nonzero coefficients for every nonzero coefficient. The VLC finds a code corresponding to the run-level pair from the code table, which is implemented with EPROM. There also exists the escape code for some run-level pairs that are not assigned to the code words in the code table (Lei and Sun, 1991). The process of the VLD is the reverse operation of the VLC. The decoding FPLD consists of two main blocks such as the VLD and the run-length decoder (RLD). The scheme of the VLD is based on the Lei s VLD architecture (Lei and Sun, 1991). Because the FPLD is not fast compared to the ASIC with which Lei implemented the VLD, we inserted the pipeline registers between the time-consuming functions that are the barrel shifter and the length decoder. The run-level pair of VLD output is decoded by the RLD block. V. CONCLUSION We developed a real-time compression/decompression system (REDS) for a nonlinear HDTV editor. It is a programmable multiprocessor architecture that uses multiple MVPs. An intrapicture coding algorithm of motion JPEG has been implemented and optimized in the REDS for a real-time operation. The HDTV image is spatially partitioned and each of the partitioned images is processed by each processing element. A field-based quantization matrix was experimentally developed, which could be used for high-quality image compression. Also, variable length encoding and VLD have been implemented in the FPLD for efficient implementation. The VLC requires sophisticated bit manipulations, which could not be efficiently implemented in a general programmable processor. The REDS demonstrated a possible architecture of a programmable real-time encoding and decoding system for a high-resolution video editor. REFERENCES B. Bhatt and D. Birks, Digital television: Making it work, IEEE Spectrum 34 (1997), ISO/IEC JTC1/SC29/WG11, Generic coding of moving pictures and associated audio, ISO/IEC 13818, ISO, (1995). J.H. Jeon, Y.S. Park, C.S. Lee, J. Kang, J.H. Park, D.S. Yoon, and H.W. Park, REDS: An encoding/decoding system for HD-VDR nonlinear editor, Proc Int Workshop on HDTV 98 (1998), pp C.L. Lee, C.S. Ho, S.-F. Tsai, C.-F. Wu, J. Cheng, L. Wang, C. Wang, Y. Hu, T. Hou, and M. Lee, Implementation of digital HDTV video decoder by multiple multimedia video processors, IEEE Trans Consumer Electronics 42 (1996), S.M. Lei and M.T. Sun, An entropy coding system for digital HDTV applications, IEEE Trans Circuit Syst Video Technol 1 (1991), S.J. Park, Y.M. Lee, K.H. Kim, Y.S. Kim, J.Y. Hwang, and K.H. Park, A video disk array for uncompressed HDTV signal, Proc of Int Workshop on HDTV 98 (1998), pp V. Ratnakar and M. Liviny, RD-OPT: An efficient algorithm for optimizing DCT quantization tables, Proc of Data Compression Conf 95 (1995), pp A. Said and W.A. Pearlman, A new fast and efficient image codec based on set partitioning in hierarchical trees, IEEE Trans Circuits Syst Video Technol 6 (1996), J.M. Shapiro, Embedded image coding using zerotrees of wavelets coefficients, IEEE Trans Signal Processing 6 (1993), SMPTE Standard, For Television Digital Representation and Bit Parallel Interface-1125/60 High-Definition Production System, (1995). Texas Instruments, Inc., Technical Report TMS320C80 (MVP), Texas Instrument, Inc., Houston, Texas, (1995a). Texas Instruments, Inc., Technical Report TMS320C80 (MVP) transfer controller, Texas Instruments, Inc., Houston, Texas, (1995b). G.K. Wallace, The JPEG still picture compression standard, Commun ACM 34 (1991), Vol. 11, (2000) 157