Hardware Platform Design for Real-Time Video Applications

Hardware Platform Design for Real-Time Video pplications. Ben titallah*, P. Kadionik**, F. Ghozzi*, P.uel**, N. Masmoudi*, P.Marchegay** *Laboratoire d Electronique et des Technologies de l Information (LETI) Ecole National d Ingénieurs de Sfax; B.P.W, 3038 Sfax, Tunisie Tel: (216)74.27.40.88 Fax: (216)74.27.55.95 uri.masmoudi@enis.rnu.tn **Laboratoire IXL-ENSEIRB-CNRS UMR-5818, Université Bordeaux1, 351 Cours de la Libération, 33 405 Talence Cedex, France {benatita;nouel;kadionik;marchegay}@enseirb.fr bstract In this paper, we present the installation of a hardware platform for video acquisition and restitution in real-time using a mixed software/hardware environment. The hardware platform is based on the ltera STRTIX development board. Besides, it is completed with a Camera interface for acquisition and a VG interface for restitution. The core of the system incorporates an IP module (Intellectual Property) of ltera Nios processor in the Quartus II development tool of ltera. During this study, we have used video sequences, which are acquired, processed and visualized while respecting temporal constraints. Keywords: FPG, VHDL, Embedded Processor, Real- Time. 1. Introduction The technology progress of integrated circuits production allows the achievement of high density such as the whole systems that can be assembled in one chip. The applications of the mobile phone, the digital TV or the video phone are examples of such integrated systems. In parallel, the increasing needs of flexibility and products fast evolution impose the integration of reconfigurable solutions, and thus the processors embedding. In fact, the use of these processors represents the best solution when the constraints of performance, cost, area and/or consumption are important. The design of a system containing one or more processors requires the use of software/hardware codesign flow [1]. This design approach consists in choosing for each functionality either a hardware implementation (that will have a little evolution) or a software programming (that will have a lot evolution or whose complexity does not allow a hardware implementation) [2]. The software part exploits the speed performance of the processor. This design methodology can be used to make a modular system of video acquisition, processing and restitution in order to facilitate the implementation of image processing algorithms. This paper is organised as follows: Section 2 presents the real time concept for the achievement of video acquisition, processing and restitution system. Section 3 describes the Nios embedded system, the Camera and VG interfaces as well as the synchronization of these two interfaces and the temporal constraints to respect. n interpretation of the Nios system implementation results is presented in Section 4. Finally, Section 5 concludes the paper. 2. Real-time video system If some processing can be executed in differed time as in the field of the data analysis, others require data processing in real time. In this case, it is a question of respecting latency constraints (The period between the input acquisition and the production of the appropriate output). The speed of processing must be compatible with the data stream of the input and output. The numerical telecommunications advent, the digital television field and the video processing have considerably amplified the requirement in real-time processing [3]. In our application of video/image processing, the chain contains modules of acquisition, processing and restitution of a video signal coming from the video source. Therefore, for us the real time is the fact that the acquisition and the processing should not introduce delay

which leads to the loss of useful data for the video restitution. The synoptic diagram of the obtained system is presented in figure 1. 8 Bits 24 Bits Figure 1. Synoptic of the video system Our video system is composed of a camera allowing the images acquisition, a processing module based on FPG (Field Programmable Gate rray) technology which integrates the Nios embedded processor and a VG monitor for the images restitution which have a 640x480 resolution in greyscale. 3. The Nios embedded system Our system design is based on the ltera Excalibur development board. The core of the board is an ltera Stratix EP1S10 F780 C6 FPG which is based on 1.5V, 0.13µm technology with a density that reaches 10570 Logic Elements (LEs), 113 KB of Embedded System Blocs (ESBs) and 427 I/O pins [4,5]. This component is optimized for the implementation of systems on programmable chip «SOPC» by integrating a softcore processor [6]. For this type of FPG component, ltera proposes a development tool (Quartus) which permits to generate the Nios softcore processor with a 16/32 Bit softcore 5 stage pipelined RISC processor and a 50 MHz operation frequency [7]. It offers a high degree of hardware configurability and customization: hardware accelerates multiplication, custom instructions, easy peripheral and custom hardware integration, etc. The term «softcore processor» corresponds to a hardware description (in VHDL or Verilog) which is synthesized then implemented on FPG. The main processing core of the Nios system, illustrated in Figure 2, is the Nios CPU. It is connected to hardware peripherals via a custom ltera valon bus. The bus has a parametric master/slave type. The parameterized valon bus interfaces are automatically generated by a special ltera Nios generating tool (SOPC Builder) for every custom peripheral integrated into the design. The Nios system for our video and image applications is composed of the following components: CPU NIOS 32 Bits. ROM for the monitor boot. 1 MB SRM. 8 MB FLSH memory. 16 MB SDRM. URT communication. Timer. VG interface. Camera interface. FLSH SDRM SRM RS232 Figure 2. Nios embedded system 3.1. The Camera interface VG interface Camera interface VLON NIOS CPU Timer URT The general structure of the camera interface is presented by the following synoptic: Clk_system master_wr Clk 14MH master_add 8 Bits Camera 32 Bits FIFO 32 Bits control DM 32 Bits LDV FDV Figure 3. Synoptic of the camera interface IRQ The camera provides in output 8 Bits video data, a clock signal with 14.318 MHz frequency and synchronization signals: line synchronization (LDV: Line Data Valid) and frame synchronization (FDV: Frame Data Valid). The camera interface permits us to send the acquired video data and other control signals towards the valon bus. This interface consists of three modules. The camera control module allows to send the acquired video data towards the FIFO module with 32 Bits word. Indeed, in the purpose of using the 32 Bits bus size totality, each four 8 Bit data pixels must be processed at 32 Bits word. The FIFO allows to memorize image line (640 pixels). It is like a buffer between the data writing and reading. The writing on the FIFO is synchronized with the Camera clock. On the other hand, the reading is synchronized with the system clock (50 MHz). Indeed, it is necessary that the reading of the FIFO data towards the SDRM is quite fast to follow the Camera stream. The third module is the DM that allows the data transfer from the FIFO towards the SDRM through the valon bus by sending «master_w», «master_addr» and «master_wrdata» signals. The cycle of writing operates until the valon bus sends «master_waitreq» signal. The entire interface is described in VHDL. It defines the interconnections of side camera as well as the connection signals with the valon bus. For compilation and synthesis, we have used the ltera Quartus tool. The obtained results are grouped in the table 1. V L O N master_waitreq B U S

Table 1. Table of the results Family LEs ESBs Freq (MHz) Stratix 593 (5%) 4 (1%) 240 NIOS CPU Camera Interface Maître VG Interface Maître We notice that the occupied area by our interface in the FPG is weak (5% of the LEs and 1% of the ESBs). The maximum operation frequency can reach 240 MHz. 3.2. The VG interface Bus valon SDRM Esclave The general structure of VG interface is presented by the following figure: V L O N B U S master rd master addr 32 Bits DM 32 Bits FIFO 32 Bits master wreq Clk_system FIFO 32 Bits Figure 4. Synoptic of the VG interface The achieved interface allows to transfer the 32 Bits data from the valon bus towards the visualization VG monitor. This interface consists of three modules. The DM module allows to transfer the data from the SDRM towards the FIFO by using «master_rd» (starting the reading of the master from the slave) and «master_addr» (addresses sent towards the valon bus) signals. buffer module is composed of two FIFO which have the same depth (640 pixels for one image line). Indeed, if the DM writs in the first FIFO, the VG controller module reads the second. This last module sends «R», «G», «B» and synchronization signals towards the VG extension board (HS: Horizontal Synchronization signal and VS: Vertical Synchronization signal) [8]. The writing on the FIFO is synchronized with the system clock (50 MHz). On the other hand, the reading is synchronized with VG clock (25 MHz). The obtained results are grouped in the table 2. Table 2. Table of the results Family LEs ESBs Freq (MHz) Stratix 840 (7%) 4 (1%) 199 We notice that the occupied area by our interface in the FPG is weak (7% of the LEs and 1% of the ESBs). The maximum operation frequency can reach 199 MHz. 3.3. Synchronization between the Camera and VG interface Our system consists of three masters (Nios CPU, Camera interface and VG interface) which can have access to the same slave (SDRM controller) through the valon bus as shown in figure 5. MUX 32 Bits VG control 24Bits HS VS Figure 5. System architecture To permit the acquisition and the restitution of the image, the Camera and VG interfaces must share the same SDRM slave. problem may happen if the VG and Camera interfaces access SDRM at the same time (Since the valon bus sends the same «master_waitreq» signal to these two interfaces). To improve the functioning of the system, synchronization is necessary between these two interfaces which is presented by the flowchart below. Start Initialization DM-VG transfer valon bus State Free valon bus DM-VG Transfer End line transfer Figure 6. Interfaces synchronization s shown in figure 6, an interface can begin the data transfer only if the other ends. In our case, the VG interface has priority since a data transfer discontinuity between the SDRM and the FIFO-VG causes problems while displaying video. For this, the DM- Camera transfer starts only when the DM-VG transfer is finished. These two DM components are described in VHDL language. They allow to accelerate the data transfer between camera, SDRM and VG interfaces. Initialization DM-Camera transfer valon bus State Free valon bus DM-Camera Transfer End line transfer FIFO-VG State FIFO-VG full

3.4. Temporal constraints The temporal constraints which should be respected to realise a real-time solution differ according to the type of application to be achieved. Our purpose is to have the maximum free time to satisfy the real-time constraints in video processing. The transfer between the FIFO and the SDRM can be realised by using the CPU. C program allows to manage and to make this transfer. temporal constraint is imposed because must empty quickly the FIFO to be aligned with the Camera stream and to present the data to the VG interface without delay. The use of the CPU reduces the transfer speed because the interruption initialization and the instructions execution of the C program are slow. Indeed, the transfer of 32 Bits word through the CPU requires a ten clock cycles. For this, we must have 768000 clock cycles for one image transfer. To solve this problem, we chose a hardware solution which consists in the realization of a DM (Direct Memory ccess) component allowing the direct data transfer between Camera interface and SDRM. This DM allows the transfer of 32 Bits word in one clock cycle. For the image transfer, we need 76800 clock cycles. In this way, the DM transfer is ten times faster than that of the CPU. The stream of data coming from the camera is of 14 Mo/s. The useful video is composed by 640x480 pixels. Since the DM transfer requires one clock cycle for each 32 Bits word, the total transfer time of an image is 1.536 ms. Therefore, the image transfer represents 4.7% of its duration by using the system clock (50 MHz) with 30 images/s camera rate. The transmitted data stream towards the monitor is 25 Mo/s. The total transfer time of one image by DM-VG is 1.536 ms. Figure 7. Vertical Video Timing (Frame) The previous chronogram represents the needed time to display one image by VG monitor which is equal to 16.7 ms. The DM-Camera transfer of one image requires 1.536 ms which represents 9.2% of the total time, in the same way for the DM-VG transfer. Thus, 18.4% of the image period would be necessary for the Camera and VG transfer and 81.6% remain free for the real time processing. 4. Implementation and results fter validation by simulation using Modelsim, compilation and synthesis by Quartus gave the following results. Table3. Table of the results Family LEs ESBs Freq (MHz) Pin E/S Stratix 5594 (52%) 31 (7%) 68.6 207 (48%) We notice that the half area of the FPG is occupied by the Nios system (52% of the LEs) whereas the utilized memory capacity is weak (7% ESBs). The maximum frequency is 68.6 MHz. This frequency is limited by the access time to the memory and delay caused by connection with all other peripherals. The implementation of the various peripherals constituting our system leaves sufficient space on the STRTIX programmable component for the addition of other IP modules and the integration of real-time video processing. 5. Conclusions In this study, we proposed a hardware platform of video acquisition and restitution in real time by using a mixed software/hardware environment. The IP modules for video acquisition and restitution are developed and based of DM module in order to interface the two external modules with the FPG component of the development board. The system implementation on the programmable component uses 48% of the inputs/outputs, 52% of the LEs (Logic Elements) and 7% of the ESBs (Embedded System Blocks). The system frequency is 50 MHz. With this frequency, 18.4 % of the image period would be necessary for Camera and VG transfer and 81.6% for the real-time processing algorithms. Our future project will consist to the integration of video processing algorithm as well as the development of real-time video applications. 6. References [1] Y. li and l Hardware-Software Co-Design of Embedded Reconfigurable rchitectures DC 2000, Los ngeles, California [2] F. Ghozzi, P. uel, Ph. Marchegay, cquisition et mémorisation vidéo sur système Excalibur, IEEE Canada, CCGEI 03, Montréal, Mai 2003.

[3] M. Finc,. Trost, B. Zajc,. Zemva, HW/SW Codesign of Real-time Video pplications Using a Custom Configurable Prototyping Platform, Electrotechnical Review, Ljubljana, Slovenija, May 21, 2003 [4] ltera Stratix Programmable Logic Device Family Data Sheet, version 2.1, august 2002. [5] D.Lewis and l The Stratix TM Routing and Logic rchitecture FPG 03, February 23-25, 2003, Monterey, California, US. [6] H. Kalte, D. Langen, E. Vonnahme,. Brinkmann,and U. Rucket Dynamically Reconfigurable System-on- Programmable-Chip PDP, January 2002, Gran Canaria Island, Spain. [7] NIOS Embedded Processor User s Guide, ltera Corporation, January 2002. http://www.altera.com/ products/devices/nios/nio-index.html [8] M. Groeneveld VG video controller for the ltera Excalibur processors Data Sheet, version 2.1, May 1st, 2003.