A -based Architecture for MPEG2 System Protocol LSIs Minoru INAMORI, Jiro NAGANUMA, Haruo WAAAYASHI, and Makoto ENDO NTT LSI Laboratories 3- Morinosato Wakamiya, Atsugi, anagawa Pref., 243-0, Japan Abstract This paper proposes a memory-based architecture implementing the MPEG2 System protocol LSIs, and demonstrates its exibility and performance. The memory-based architecture implements the full functionality of the MPEG2 System protocol for both multiplexing and de-multiplexing MPEG2-encoded streams. It consists of a core CPU, memories, and dedicated application-specic hardware. It is designed and optimized by hardware/software co-design techniques. The LSIs provide sucient performance and exibility for real-time applications of the MPEG2 System protocol. Introduction Recently, the MPEG2 standard has emerged as a method for eectively compressing video and audio streams, while maintaining their quality. Since the MPEG2 standard aims at retaining high quality, many transmission and storage applications are being considered : satellite broadcasting or cable TV (transmission), and CD-ROM (storage). A typical MPEG2 environment consists of video and audio encoders, video and audio decoders, a multiplexor (MUX), and a de-multiplexor (DMUX). The \MPEG2 Video" standard [] species the coded representation of video and the decoding process, while the \MPEG2 Audio" standard [2] does the same for the audio. The \MPEG2 Systems" standard [3] species the system layer of the coding, and denes two protocol levels : the Transport Stream (TS) used in the environments where errors may occur ; and the Program Stream (PS) used in the errorfree environments. The TS (resp. PS) species the format of a multiplexed stream consisting of several audio and video MPEG2 streams over lossy (resp. lossless) physical channels. To implement the MPEG2 CODEC systems eciently, the development of key component LSIs was desired. Video encoder [4, 5] and decoder LSIs [6, 7] meeting the \MPEG2 Video" standard, and audio encoders [8] and decoders LSIs [9] meeting the \MPEG2 Audio" standard have been already developed. However, neither MUXs nor DMUXs for both the TS and PS meeting the \MPEG2 Systems" standard are developed, except for the TS DMUX [0]. There are two problems linked to the development of full-functionality MPEG2 System MUX and DMUX : the complexity of the MPEG2 System protocol, and the fact that it is necessary to develop the LSIs concurrently with the standard (tracking a moving target). ecause of the protocol complexity, a fullcustom implementation of MUX and DMUX will lead to huge and complicated state machines : The syntax of both TS and PS greatly depends on the elementary stream. Therefore, the MUX must have enough performance to analyze and multiplex the elementary stream, while the DMUX must have enough performance to de-multiplex and reconstruct the elementary stream. Real-time requirement limits the MUX and DMUX design to a one-pass analysis of the elementary stream, TS or PS, in spite of their complicated syntax. Since the operations for the TS are dierent from the ones for the PS, it is dicult to implement both procedures on the MUX and DMUX LSIs. ecause of the concurrent development of the LSIs and the standard : ED&TC 96 0-8979-82/96 $5.00 996 IEEE
The LSIs must have sucient exibility to track easily the modications of the standard specication. Video Camera Microphone A/D A/D Video Encoder Audio Encoder Multiplexer (MUX) Network It is dicult to implement all the functions with only dedicated application-specic hardware. To solve both problems (complexity and moving target), we propose a new memory-based architecture for LSIs implementing the MPEG2 System protocol. The architecture supports the full functionality of the \MPEG2 Systems" standard, for both the MUX and DMUX, and for both the TS and PS. The architecture consists of a core CPU, memories, and dedicated application-specic hardware designed and optimized using hardware/software (HW/SW) co-design techniques. The software on the core CPU provides the required exibility and the dedicated applicationspecic hardware supports the performance. These LSIs based on this architecture provide sucient exibility and performance for real-time applications of MPEG2 System protocol. Structure of the paper. Section 2 describes the memory-based architecture for MPEG2 System protocol LSIs and the HW/SW partitioning with this architecture. Section 3 presents the implementation of the MUX and DMUX and their fabrication. Section 4 presents the results of an evaluation of the memorybased architecture by implementing the MUX and DMUX LSIs. 2 -based Architecture 2. MPEG2 System Protocol Processing 2.. An MPEG2 CODEC environment An example of an MPEG2 CODEC system is shown in Figure. The video (resp. audio) encoder encodes video (resp. audio) into elementary streams. The MUX multiplexes one or more elementary streams of video, audio and user into one program and transmits it on the network. The DMUX extracts one program from the network and de-multiplexes one or more elementary streams of video, audio and user from it. The video (resp. audio) decoder decodes the video (resp. audio) elementary stream into the video (resp. audio). Monitor Speaker D/A D/A User Data Video Decoder Audio Decoder User Data De-multiplexer (DMUX) Coder Decoder Network Figure : An example of MPEG2 CODEC System -header TS-header PSI PTS DTS PTS DTS payload 88[byte] payload Elementary Stream Transport Stream output MUX DMUX input output Figure 2: Composition of Transport Stream -header pack-header payload directory packet Elementary Stream Program Stream input MUX DMUX input output output Figure 3: Composition of Program Stream input 2..2 Hierarchical Packetizing and Data Dependency A hierarchical packetizing scheme is used in the MPEG2 System protocol. First, the elementary stream from the encoder is packetized to the Packetized Elementary Stream (). Next, the is further packetized to the TS or PS, depending on the application. As shown in Figures 2 and 3, the elementary stream from the encoder is written down in the payload ( packet bytes), and the itself is written down in the TS or PS payload. Some values in the header are determined through the analysis of the elementary stream. Presentation Time Stamp (PTS) and Decoding Time Stamp (DTS) [3]
which are used to synchronize the video with the audio, are examples of these values. The TS/PS syntax depends on the elementary stream, which complicates the packetization process. 2..3 Required Performance of MUX and DMUX The \MPEG2 Systems" standard does not specify the bit rates of video, audio and user. In our design (MP@ML []), the maximum bits rate of video, audio and user are set to 5[Mbit/s], 384[bit/s], 92[bit/s], respectively. They are decided from the real-time use of MUX and DMUX. The hierarchical packetizing and the analysis of the elementary stream which must be done in real-time, increase the complexity of the MUX/DMUX architecture. 2.2 -based Architecture by HW/SW Co-design Techniques 2.2. Design Trade-o between HW/SWoriented Models The design ow for selecting the memory-based architecture to implement MPEG2 System protocol LSIs is shown in Figure 4. In this gure, the functions enclosed by bold (resp. dotted) lines are implemented in hardware (resp. software). Figure 4(a) is the hardware-oriented model. Its implementation is faithful to the specications and its performance is high because of the distributed and parallel processing. However, it has many resource parameters (such as the number of FIFOs and their memory sizes), and as all the functions are implemented in hardware, the exibility is poor, and the design is complex. y contrast, Figure 4(b) is the software-oriented model (a) Hardware-oriented Model (b) Software-oriented Model in FIFO m FIFO n out Design Trade-Off in m n out Syntax Controller - Packetizing - Analysis High Low Performance Low High Flexibility Read Cont. Core CPU Write Cont. Syntax Cont. - Packetizing - Analysis Advantages: - Faithful to the specifications - High performance Disadvantages: - Many resource parameters - Low flexibility - Complicated construction No Partitioning into HW/SW Flexibility Performance Gate Size Yes Advantages: - Few resource parameters - High Flexibility - Controllers (Read/Write/Syntax) are controlled by the software Disadvantage: - Performance depends on the memory -based Architecture for MPEG2 System Protocol LSIs Figure 4: Derivation of -based Architecture from HW/SW-oriented Model
m n in Write Cont. uffer (2-port) Read Cont. out uffer Port A Port (a) Software-oriented Model Data Input Analyze Read Write Packetize Core CPU (2-port) Syntax Cont. - Packetizing - Analysis Port A Port (b) -based Architecture Data Output Figure 5: -based Architecture for MPEG2 Systems similar to protocols [] implemented on a workstation. It has few resource parameters (such as only memory size), and all the functions are controlled by the software. The exibility isvery high, but the performance greatly depends on the response time of the memory used. 2.2.2 -based Architecture for MPEG2 Systems Taking into account the constraints of performance, exibility and gate count, the memory-based architecture is selected. It represents a middle ground between the hardware- and software-oriented models as shown in Figure 5. It features : () a hardware controller and a buer (top part of Figure 5) ; (2) a software controller (core CPU) and a memory (bottom part of Figure 5). Toachieve the performance, The memory and the buer are separate. The hardware controller uses the buer for I/O, while the core CPU uses the memory for the syntax analysis. The buer and memory have the dual ports. oth the hardware controller and the core CPU can access both buer and memory at the same time. This memory/buer distribution makes it possible to introduce pipelined processing shown in Figure 6. For both the MPEG2 System MUX and DMUX, the memory-based architecture features a three step pipeline. We explain the pipeline of the MUX architecture. In the rst step, the hardware controller Figure 6: Pipelined Data Processing receives the elementary stream in the buer. In the second step, the core CPU transfers the packet from buer to memory, computes the internal state of the and TS headers, and writes the back to the buer. In the third step the hardware controller sends the packetized from the buer. This pipelined scheme hides almost all core CPU cycles (analyzing and packetizing) behind the I/O cycles. Moreover, I/O cycles are also overlapped each other. Figure 6 shows the dierence between the processing under the software-oriented model and this architecture. In the proposed architecture, the hardware controller and the core CPU use the distributed buer and memory eectively, which improves the throughput up to the performance required to MPEG2 System protocol LSIs. This architecture features the good performance of the hardwareoriented model and the high exibility of the softwareoriented model. 2.2.3 Detailed Partitioning of HW and SW The detailed partitioning of the hardware and the software for this architecture shown in Figure 5 is described here. The jobs of the write controller in Figure 5 are () the input of, (2) header search, (3) clock generation for the video, (4) the synchronization of the TS and (5) 32[bit] CRC calculation. The read controller does (6) the output of. The (7) buer control, (8) scramble control and (9) the synchronization of System Time Clock (STC) are done by both write and read controllers. The core CPU processes (0) the syntax analysis, and does () rate calculation for video, (2) the synchronization between video and audio, (3) the Program Specic Information (PSI) generation.
0-3 [s] 0-6 () Input of Data (2) Header Search (3) Clock Generation (4) Synchronization of TS (5-) 8[bit] CRC Calculation (6) Output of Data (7) uffer Control (8) Scramble Control (9) Synchronization of STC - Hardware -Software (0) Syntax Analysis () Rate Calculation (2) Synchronization of Video and Audio (3) PSI Generation (5-2) CRC Calculation Control Figure 7: Response Time Figure 7 shows the required response time of the above functions. Considering the real-time use, 0 03 [s] is the criteria for implementing the functions in the hardware or the software with exception of 32[bit]-CRC calculation. A 32[bit]-CRC calculator is not available because of the gate size, so it is realized with an 8[bit]-CRC calculator controlled by the software. 3 Implementations of MUX and DMUX LSIs 3. Design Approach The full functionality of the MUX and DMUX for both the TS and PS have been implemented on the proposed memory-based architecture. To shorten the design turn-around-time (TAT), we adopt the following approach. The hardware parts common to the MUX and DMUX LSIs are extracted. The hardware block commonality enables to shorten the design time and improves software productivity. The common blocks are the core-cpu, the host processor interface, the memory and the memory interface, and the HW/SW interface at the core-cpu part. The gates in MUX and DMUX LSIs are fully synthesized from the specications written in hardware description languages (HDL). The core-cpu is written in SFL [2], and the remainder is written in Verilog-HDL [3]. The Core CPU is a 32[bit] RISC processor like DLX [4]. The C-compiler and the assembler were developed with it and then all the software was described in C-language. Communication between the software and the hardware is accomplished through the HW/SW interface registers using the memory mapped I/O technique. 3.2 Fabricated LSIs 3.2. Functions of MUX and DMUX LSIs The main functions of MUX and DMUX are listed in Table. The MUX and DMUX LSIs support the full functionality of the MPEG2 System protocol. 3.2.2 Features of MUX and DMUX LSIs Table 2 shows HDL specication line count and the synthesized gate count. There are 2086 lines of HDL specication shared by the MUX and DMUX LSIs, except for the core CPU specication, which is 3995 lines in SFL. Table : Implemented Functions Input of Data Output of Data Synchronization of STC [3] Synchronization of Video and Audio Prole and Level [] MUX DMUX Elementary Stream (Video, Audio, User Data) TS or PS Elementary Stream TS or PS (Video, Audio, User Data) PCR or SCR PTS and DTS MP@ML
Table 2: Lines of HDL and Gate Count MUX DMUX Core-CPU Lines of HDL 055 (Verilog-HDL) 482 (Verilog-HDL) 3995 (SFL) Gate Count 7[gate] (including Core-CPU) 73[gate] (including Core-CPU) 27[gate] 0 Table 3: Lines of Software Lines of C-Lang. Lines of Assembly-Lang. MUX 720 2242 DMUX 877 9056 Figure 9: Photograph of MUX Table 4: Features of MUX and DMUX 7[gate] + R.F. + Clock + MUX 8[byte] DPRAM 73[gate] + R.F. + Clock + DMUX 8[byte] DPRAM Technology 0.5[m] CMOS Embedded Gate Array Clock 27[MHz] Size 4[mm] 2 4[mm] Package 304 pin QFP Voltage 3.3[V] Power.85[W] Table 3 shows the size of the C-language code running on the core CPU and the size of the assemblylanguage code generated by the C-compiler. The C-language line count is about half the assemblylanguage line count, which indicates that the productivity of the software improves at least by a factor of two by using the C-language. Table 4 lists the features of the MUX and DMUX LSIs. Figures 8 and 9 show the oorplan of the MUX and DMUX LSIs and a photograph of MUX LSI, respectively. The MUX and DMUX LSIs feature the same oorplan in order to shorten the design time. The rate of the commonality between MUX and DMUX reaches 60% in terms of the chip area. CoreC PU Clock Clock M UX/DM UX RF 4 Evaluation 4. Methodology Local HostIF ufferdpram Data DPRAM Figure 8: Floorplan of MUX and DMUX To validate and evaluate the functionality and performance of the MUX and DMUX LSIs, a software system simulating both circuits (software MUX/DMUX) written in C-language was developed [5], and some typical streams such as elementary streams, TSs and PSs were processed on it. The experiments using these streams were also done on a prototype evaluation board with the MUX and DMUX LSIs.
Video Video Table 5: Average CPU Cycles Per One[byte]-Transmission System MUX(TS) MUX(PS) Software MUX 3375 627 MUX LSI 0 0 Improvement 2 337 2 62 System DMUX(TS) DMUX(PS) Software DMUX 276 375 DMUX LSI 0 0 Improvement 2 27 2 37 Audio User Data PSI, User Data, ITU-T TS Header for Video TS Header for Audio TS Header for PCR TS Header for Null Reserved (a) Map for TS Audio User Data User Data, ITU-T Header for Video Header for Padding Header for Audio Pack Header, System, PSM directory Packet (b) Map for PS Figure 0: MUX's Maps for TS and PS 4.2 Results The behavior of MUX and DMUX LSIs on the board were completely the same as those of the software MUX/DMUX. The MUX and DMUX LSIs' throughput was 2.6[Mbit/s] which is.4 times of the requirements for these LSIs. The results also shows that the MUX and DMUX LSIs support the full functionality of the MPEG2 System protocol. oth chips can support one video stream and either two audio streams or one audio and one user simultaneously. It is dicult to evaluate the exibility with the memory-based architecture quantitatively. The MUX and DMUX LSIs were implemented on the same proposed architecture easily. Moreover, both the TS and PS were also implemented on these LSIs by the software completely. The software and the applicationspecic I/O buer design for TS and PS enable the full functionality of the MUX and DMUX. Figure 0 shows the MUX's I/O buer maps for the TS and PS. For the TS, the TS header is saved in the I/O buer, while for the PS, one header and several PS headers are saved. The same device is applied to the the buer used in DMUX. In this way, the memory-based architecture has enough exibility for MPEG2 System protocol LSIs. To evaluate the performance of the memory-based architecture, the average CPU cycles per one[byte]- transmission of the software MUX/DMUX and those of the MUX and DMUX LSIs are shown in Table 5. It is reduced to form /00 to /300. In other words, the MUX and DMUX LSIs accomplish more than onehundred times performance improvement, compared to the software MUX/DMUX which can be regarded asatypical software-oriented model. These results demonstrate the exibility and performance of the memory-based architecture in implementing the full functionality of the MPEG2 System protocol. These LSIs featuring the memory-based architecture provide sucient performance and exibility for real-time applications of MPEG2 CODEC Systems[6]. 5 Conclusion This paper proposes a memory-based architecture for the MPEG2 System protocol. It demonstrates the exibility and performance of the memory-based architecture. oth MUX and DMUX LSIs support the full functionality of the \MPEG2 Systems" standard. The memory-based architecture consists of a core CPU, memories, and dedicated applicationspecic hardware, which is decided upon and optimized using HW/SW co-design techniques. The LSIs provide sucient performance and exibility for real time applications of the MPEG2 System protocol. In the near future, we will study the performance limitation of the architecture, and investigate ways to expand it to MP@HL [] for HDTV. We will parameterize the core CPU, the HW/SW interface and the host processor interface, and we will try a new system-level synthesis.
Acknowledgements The authors would like to thank Dr. Osamu aratsu and Tamio Hoshino of the NTT LSI laboratories for supporting this work. Thanks are also due to Takaaki Izuoka of the NTT Human Interface laboratories and the members of the Advanced LSI laboratory for useful discussions. References [] Video - Generic Coding of Moving Pictures and Associated Audio ISO/IEC 388-2 International Standard., November, 994. [2] Audio - Generic Coding of Moving Pictures and Associated Audio - ISO/IEC 388-3 International Standard., November, 994. [3] Systems - Generic Coding of Moving Pictures and Associated Audio - ISO/IEC 388- International Standard., November, 994. [4]. Ishihara et al. A half-pel precision mpeg2 motion-estimation processor with concurrent three-vector search. IEEE International Solid- State Circuits Conference digest of technical papers, pp. 288{289, 995. [5] T. ondo et al. A two-chip realtime mpeg2 video encoder with wide range motion estimation. Symposium Record HOT Chips VII, pp. 95{0, 995. [6] T. Demura et al. A single-chip mpeg2 video decoder lsi. IEEE International Solid-State Circuit Conference, pp. 72{73, 994. [7] M. Toyokura et al. A video dsp with a macroblock-level-pipeline and a simd type vectorpipeline architecture for mpeg2 codec. IEEE International Solid-State Circuit, pp. 74{75, 994. [8] Texas Instruments. Tms320c3x users manual. 994. [9] Texas Instruments. Tms320avx users manual. 993. [0] C-Cube Microsystems. Data sheet of cl90 - transport layer demultiplexer. 994. [] Douglas Comer. Internetworking with TCP/IP: Principles, Protocols, and Architecture. Prentice- Hall, Inc., 988. [2] Y. Nakamura. An integrated logic design environment based on behavioral description. IEEE Transaction on CAD, Vol. CAD-6, No. 3,, 987. [3] Open Verilog International. Verilog HDL Reference Manual (LRM). 99. [4] J. L. Hennessy and D. A. Patterson. Computer Architecture A Quantitative Approach. Morgan aufmann Publishers, Inc., 990. [5] Software MUX/DMUX System (Ver.0). NTT/NEL/NTTS Internal Reports, 994. [6] Yutaka Tashiro et al. Mpeg2 video and audio codec board set for a personal computer. IEEE Global Telecommunications Conference, Vol., pp. 483{487, 995.