Bachelor Thesis. Augmented Reality using a Virtual Reality. R.J.A. Blokker & L.M. Noordam. Implementation on an FPGA device

Transcription

1 Bachelor Thesis Augmented Reality using a Virtual Reality setup R.J.A. Blokker & L.M. Noordam Implementation on an FPGA device

2

3 Augmented Reality using a Virtual Reality setup Implementation on an FPGA device by R.J.A. Blokker & L.M. Noordam Date: June 15, 2016 Student numbers: L.M. Noordam: R.J.A. Blokker: Project duration: April 18, 2016 July 1, 2016 Thesis committee: Prof. Dr. Koen Bertels TU Delft, chair Prof. Dr. Elmar Eisemann, TU Delft, supervisor Dr. Jelena Popovic, TU Delft, jury member

4 Abstract Augmented reality is a very new market and at moment only available for early adopters. The available products are expensive and still under development. Furthermore, the current products can only augment a small percentage of the user s field of view. The product described in this thesis uses equipment available on the consumers market to implement augmented reality. By connecting two cameras to an Oculus Rift and some additional hardware, an environment is created in which the users can experience augmented reality with virtual reality equipment. The additional hardware consists of an FPGA and a personal computer taking care of adding content to the virtual image of the real world. Directly connecting the cameras to the FPGA results in a short low latency path. This should allow the user to perceive the world with the least possible amount of introduced lag. The prototype is not finished yet, so whether it s possible to achieve the low-latency throughput is still not known. However, the modules that are responsible for reading from an HDMI input, writing to an HDMI output, controlling the memory and conversion between color spaces have already been implemented. The details of these implementations are discussed in this document. We hope you enjoy reading our thesis, R.J.A. Blokker & L.M. Noordam Delft, June 2016 ii

5 Contents Abstract 1 Introduction 1 2 Program of Requirements Introduction Functional Requirements System Requirements Preliminary Design Process Design Process Draft Concepts FPGA System Overview Introduction HDMI Camera Multiplexer Buffer USB Connection Implementation HDMI Camera decoder Memory Interface AXI-lite Reader System Interconnects Validation Sync Generator Camera Decoder AXI-lite Reader Discussion of Results 18 8 Conclusion and Recommendations Conclusion Recommendations Appendix 20 A VHDL Source Code 20 A.1 Sync Generator A.2 Camera Decoder A.3 AXI-lite Reader Bibliography 31 ii iii

6

7 1 Introduction The virtual reality market is upcoming. More and more consumer versions of virtual reality (VR) glasses become available. Augmented reality (AR) is somewhat like VR. With AR not the whole image is virtual, but only certain parts. The main part of what a user sees is the real world. The user can see in front of him just like without the glasses. Extra attributes are added to the image to make it augmented reality, just like on the cover image [1]. One of the goals of this project is to make AR glasses from a VR glasses. Such an approach has been attempted before by others, usually involving a computer sending camera images to a VR glasses [2]. Using a computer introduces too much delay for a good working system. When there is too much delay, collisions with objects in the room or a wall are possible. This is because the place you think you are is not the place you really are. Grabbing objects is also harder because of the delay. The user experiences a delay in the movement of his arm. Therefore reaching to an object has to be done really slow to prevent the user to push the object over. There are AR glasses available to deal with the problem of delay. These devices have seethrough lenses. One of them is the Microsoft HoloLens [3]. These devices are still very expensive and in development. They also exhibit a small field of view. This project responds to this problem. With this system affordable AR glasses become possible. By using a consumer version of VR glasses in addition with our system the total cost will be much lower than buying already existing AR glasses. By adding a fish-eye lens to the cameras, the field of view will be higher than with a HoloLens. In the next chapter (Chapter 2) the requirements are discussed, what has to be accomplished to reach the goal of the project. In Chapter 3 the preliminary design process is discussed, what choices were made. In Chapter 4 an overview of all components in the design are discussed. Chapter 5 describes how all the components are implemented. In Chapter 6 the components that were designed are validated, to show how they work. After the validation the results will be discussed in chapter 7. Finally in Chapter 8 the conclusion and recommendations for future projects. 1

8 2 Program of Requirements 2.1. Introduction The proposal [4] requested to build an environment in which augmented reality is realized using the cheaper virtual reality equipment available on the consumer market. While solutions for creating AR already exist, the market for VR equipment is a few years more mature. And because the two markets have a similar end product, it makes sense to do research that might result in more integration between those markets. The final product will be used by developers building applications for AR systems. Therefore it should be easy to integrate the new product with existing electronic devices Functional Requirements The following statements describe the purpose of the product to be developed: [2.1] The system must be able to read the video output stream of another device using a commonly known standard for transferring video streams. [2.2] The system requires an interface with at least two cameras to capture reality in stereo vision. [2.3] The system should be able to merge and synchronize the incoming video streams into one output stream. Decision making is based on an predefined video input stream. [2.4] The system should be able to output a video stream in a format that can be used as input for an existing VR device. [2.5] The system must be able to output the camera video streams using a technology that enables high bandwidth video transfers over a single cable connection. [2.6] The lag introduced by the AR system should be as low as possible. The path from the cameras to the video output has the highest priority in this aspect. 2

9 2.3. System Requirements System Requirements Elaborating on the functional requirements from the previous section results in: [3.1] The camera interface will be determined by the purchased type of camera. The output signal type must consist of raw data as this allows for easy further processing. However, this will increase the need for a high internal system bandwidth. [3.2] A usable generic video transport protocol, supported by a lot of consumer devices, is the HDMI specification [5]. This interface allows for a high resolution video stream and can be decoded into readable data quite easily. [3.3] Decision making about which input stream to use as output steam is based on the generic video input stream on per pixel basis. When a pixel of the generic input video stream has a color value equal to RGB value (0,255,0), the corresponding camera pixel should be written to the video output instead. [3.4] An FPGA as interconnect between all components generating or consuming video streams. The FPGA requires an HDMI in/out connection to process the generic video stream input and generate the video output to the VR device. Also at least 64 GPIO pins are required for communication with the cameras and exporting this video stream. [3.5] As VR device the Oculus Rift DK1 is used. It has an HDMI input port and displays the video stream with a refresh rate of 60 fps. The FPGA should be capable enough to output the required frame rate. [3.6] The latency from the camera video out to the Oculus Rift HDMI port should not be larger than 50 milliseconds. The reason is that the user of the product should be able to wear the Oculus Rift without interruption for at least half an hour without getting motion sickness. [3.7] For exporting the video stream of the cameras the USB 2.0 protocol is used. This is not ideal, since the effective throughput of around 280 Mbps [6] is limited. However, it should be sufficient for the purpose of transferring video data consisting of several frames per second.

10 3 Preliminary Design Process 3.1. Design Process In the first few weeks of the project, most of the time was spent on having brainstorm sessions with the team. A few different options were considered for implementation and this chapter is meant to explain the process we went through before purchasing the required hardware. Having the team agree about one of the solutions to work with took longer than expected. Augmented reality is based upon creating additional layer of information on top of how we perceive the world around us. This is different from virtual reality, because that is meant to immerse the user into a non-existing world. In order to implement augmented reality into a virtual environment, the real world has to be recreated into the virtual one. This can be done by adding two cameras to the system to replace the user s eyes. By connecting the virtual reality equipment and the cameras to a powerful rendering computer, an AR experience can be created using VR equipment. However, the solution using a computer to connect all the components has already been done before and therefore cannot be categorized as state-of-the-art technology [2]. One of the disadvantages involving a computer as main calculator is that the system adds a lot of latency to the virtual environment. When the user wants to move around, the recreated world the user perceives will lag behind. The solution to this problem is to create a low-latency path between the camera video streams and the virtual reality equipment. This can be done by using an FPGA, because the device is able to process video streams (requiring a relatively high bandwidth) in real time while keeping the added latency low Draft Concepts To make the project more interesting, the initial idea was to use four cameras recording two different spectra. Two cameras are used for the stereoscopic view and the other two are used as cheap alternatives for an infrared camera. Modifying a normal camera into an infrared camera is possible, because the embedded sensor is capable of perceiving frequencies in the near infrared spectrum [7]. However, because the human eye cannot perceive infrared, these relatively low frequencies are normally suppressed by a filter placed in front of the lens. By removing this filter, the camera can be used for recording near-infrared data. After deciding on which hardware components to use, three options for implementing the design were discussed. At this moments the team was already divided in three subgroups: Computer, Camera and FPGA. The first design proposal can be found in Figure

11 3.2. Draft Concepts 5 Figure 3.1: First discussed implementation design In Figure 3.1, the four cameras are connected to the FPGA through their video interfaces and the FPGA is connected to the computer using USB 2.0 and HDMI. For each pixel in the output stream can be chosen whether to use the IR camera or normal camera video stream. This decision would be made based on a hard-coded threshold value. Because the decision making is done on the FPGA, the added latency will be relatively low. However, more code needs to be written to program the FPGA, which is more difficult than writing code for a personal computer. Also, when the user would want to process the IR camera video streams on the computer, this data has to be send though the USB 2.0 interface, which has a limited throughput. Figure 3.2: Second discussed implementation design Compared to Figure 3.1, the second implementation depicted in Figure 3.2 adds a serial control component from the computer to the FPGA. By using this concept the computer is able to decide which pixels should be selecting from the available video streams. The FPGA can use these decisions as a mask for selected the different video input streams. This adds flexibility to the system. However, the problem of the limited USB 2.0 connection is still not solved. Complexity of the FPGA design is also increased by moving more logic into the device. The schedule depicted in Figure 3.3 has the advantage of being modular. Any device which

12 6 3. Preliminary Design Process Figure 3.3: Third implementation sketch can be connected to a computer is able to interface with the augmented reality setup. Also the latency of the camera video streams is low, because the required amount of processing time on the FPGA is at most a few frames. The downside of this approach is the extra lag introduced in the augmented reality layer. However, because creating objects in a virtual environment based on the real world requires complex computations, this is not something that should be done on the FPGA FPGA After discussing the top level overview of the preliminary design the team had to decide on which FPGA would be used for the project. The top level design implies the following requirements for the FPGA: An HDMI input port for transporting video from the computer to the FPGA An HDMI output port for transporting the video stream to the Oculus Rift At least 64 general-purpose input/output (GPIO) pins for connecting the four cameras An USB or serial interface fast enough for sending video streams to the computer After consulting with the University of Delft the team found out they did not have an FGPA available suiting the requirements. So the search was continued on the internet. This was a lot harder than expected, because the FPGAs that were found almost always had a price higher than the available budget. After intensive research two suitable FPGAs were found: The Terasic Cyclone V GX Starter Kit, 200, found at Mouser Electronics [8] The Nexys Video Artix-7 FPGA, $490 found at the Digilent webstore [9] The Nexys board seems to be twice the price of the Terasic. However, Digilent offers an academic discount, which results into the Nexys board having a price around 300 (shipping included). This still is quite a difference, although it can be explained by noticing the Nexys board is one of the cheapest available devices having an HDMI sink. The Terasic board is shown in Figure 3.4 and the Nexys board can be found in Figure 3.5. Both images were copied from the websites cited in the previous bullet points.

13 3.3. FPGA 7 Figure 3.4: The Terasic Cyclone V GX Starter Kit Figure 3.5: The Nexys Video Artix-7 FPGA The problem here is that both boards don t have enough GPIO pins available to interface with the cameras. Luckily enough there are expansion boards available to increase the amount of GPIO pins. The advantage of the Terasic board is the lower price and the cheaper expansion board of only 55 [10]. The cheapest available FMC expansion board suited for the Nexys Video has a ridiculous price tag of 180 [11]. The expansion boards are shown in Figure 3.6 and Figure 3.7. Images were copied from the previously mentioned citations. Figure 3.6: Expansion Board for the Terasic FPGA Figure 3.7: Expansion Board for the Nexys Video FPGA The disadvantage of the Terasic board is that it doesn t have an HDMI sink and therefore another interface is required to connect one of the computer s video outputs to the FPGA. After consulting with our supervisor about the price of the boards and their capabilities, the decision was made to buy the Nexys Video board and corresponding expansion board. Although these two components together are twice the price of the Terasic and its expansion board, the Nexys Video is a better option. The main reason for choosing the Nexys Video is because the prototype will have an HDMI input port. This results into the product being reusable for other projects developing solutions using augmented reality. Also a lot of documentation about the Nexys Video is available on the website of Digilent, which will ease the development [12].

14 4 System Overview 4.1. Introduction To be able to send the camera feed as fast as possible to the VR glasses a Field-Programmable Gate Array (FPGA) is used. That way, the camera feed doesn t have to go to a computer where software is used to decode the image and encode it to send the image to the VR glasses, because that is the cause of the biggest delay. In the FPGA this is done without software, only with hardware. This way the delay will be at most a few frames. The FPGA connects all parts of this project. The FPGA gets the camera feed, sends it to the computer, gets video input from the computer for the AR part, mixes the camera feed with the AR video stream from the computer and sends it to the VR glasses. To program the FPGA Vivado [13] used. Vivado can deal with Verilog and VHDL. In this project the language used is VHDL. In Vivado a design can be made by converting VHDL code to a block with in and output ports and connecting them with wires. Vivado also works with Intellectual property (IP) blocks, these are blocks with a certain function that can be placed inside the design and adjusted to a certain level. The real VHDL or Verilog code can not be adjusted. These blocks helped to achieve certain goals, that will be specified below HDMI The FPGA has two HDMI ports, one input port and one output port. To send video signals over the HDMI channels the protocol had to be investigated [14]. HDMI works with Transition Minimized Differential Signaling (TMDS). TMDS [15] uses four lines to transfer video data. Three lines for each one color, Red, Green and Blue and one line with a pixel clock signal. An IP block was found that could decode TMDS to RGB values, a pixel clock, Horizontal Sync (HSync), Vertical Sync (VSync) and a video enable signal [16] and one that could decode those signals to TMDS. The pixel clock has a rising edge during every pixel. HSync is high after a horizontal line is finished. VSync becomes high after a whole frame. Video enable is high during the part that pixels are send. To make a valid HDMI video signal, the right pixel clock, HSync, VSync and video enable signal had to be produced. To get these signals a modeline calculator was used. A modeline calculator calculates at what time the HSync and VSync signal have to be high. HDMI has a front and back porch after each horizontal line of pixels. That is the reason that there is a video enable signal. This makes sure that non-valid data in the front or back porch is not seen as a pixel, because the signal will be low there. After the front porch the HSync will be 8

15 4.3. Camera 9 high for a certain time. After that is the back porch. After the back porch a new horizontal line of pixels is send. Until the frame is full. At that time the VSync will be high after his front porch and the screen will then know that this was the end of the frame Camera The camera outputs its pixels in YUV format. Y stands for the amount of lux, how much light there is in a pixel. U and V are the chrominance (color) components. Because the TMDS block outputs RGB values, YUV is converted in the FPGA to RGB values. The camera also sends a VSync signal and a HREF signal, that looks a lot like a video enable signal that HDMI uses. For further details about the implementation of the camera, see Augmented Reality in a Virtual Reality Setup: The Camera [17] Multiplexer To be able to make the glasses AR there must be a way to mix the camera feed with the video images from the computer. To this extent, a solution similar to green screening is chosen (Chroma Keying [18]). The HDMI output of the computer will have pixels with an RGB color value of (0,255,0). In the FPGA these pixels will be replaced by the corresponding pixels from the video camera. The biggest advantage of this implementation is the short path between the camera and the Oculus Rift Buffer In order to synchronize the camera feed with the video stream of the computer a buffer is needed. This is because both feeds will not be at the same pixel at the same time. Both feeds have to be buffered in the DDR3 RAM available on the FPGA. This will unfortunately cause extra delay, but it is necessary for the system to work properly USB Connection To be able to send the camera feed to the computer there is chosen to use the USB connection on the FPGA. This part is done by another group that was looking into the computer part of the project [19]. The camera feed is needed in the computer to be able to make the system AR. Objects can be placed upon the camera feed, after which they send only their overlay via HDMI. This is later on mixed with the camera feed in the Mux part of the design.

16 5 Implementation In this chapter the implementation of all subsystems are discussed HDMI HDMI needs HSync and VSync singals at certain times. To determine at what time they had to be high the code from appendix A.1 is used. In a block design, this code will look like Figure 5.1. Here, you can see the input ports on the left and the output ports on the right. The clk (clock) input needs to be the pixel clock, which is generated somewhere else. It has a reset port to reset all signals. The sw port are 7 switches that can be connected to this version of the block. These 7 switches are converted to a color, this is for debugging only. The RGB input consist of 24 bits, which is equal to 3 bytes. One byte per color, red, green and blue. The last input, btn, is a button that is used for debugging. Pressing the button switches the output between the color made by the switches or the RGB value inputted. sync_generator_0 clk resetn sw[7:0] RGB[23:0] btn vsync hsync venable displayout[23:0] sync_generator_v1_0 Figure 5.1: Sync Generator At the output there are four ports. vsync becomes 1 after a whole frame is send. hsync becomes 1 after one horizontal line of pixels is sent. venable becomes 1 when the block is sending pixels. displayout is the RGB value of the current pixel. Inside the block are two counters, counterx and countery. counterx increases every rising edge of the clock by one. When counterx has been reset, and thus is equal to zero, venable becomes 1. When the counter is equal to the width of the frame, for example 1280 pixels, venable becomes 0. After another few clock cycles, the front porch, hsync becomes 1. The amount of clock cycles needed are calculated with a modeline calculator. After another few clock cycles hsync becomes 0 again. Then there is a back porch for a few clock cycles, 10

17 5.2. Camera decoder 11 after which counterx returns to 0, and countery is increased by one. This way the HDMI controller in a screen knows a line is done and the next line will start. After countery reaches the height of the image, for example 1024 pixels, venable will not become 1 anymore, but there will still be HSyncs. After the front porch of the VSync, vsync becomes 1. After a certain amount of horizontal lines vsync becomes 0 and there will be a VSync back porch. After the back porch the whole frame is done and the counters will be reset to zero again Camera decoder To read out the pixels from the camera fast, YUV422 is used. YUV422 means that two bytes with Y values of two adjoining pixels are sent and one byte U and one byte V that are the average for both pixels. This way two pixels can be recovered from four bytes. Consequently, the pixel clock of the camera can be higher. Because other parts of the project use the RGB format YUV has to be converted to RGB, which is done with three formulas [20]. The next formulas are used: R = Y (V 128) (5.1) G = Y (U 128) ( (V 128)) (5.2) B = Y (U 128) (5.3) The block design can be found in Figure 5.2. The inputs are HREF, VSync, PixelClk, resetn and data. All of these inputs, except resetn, come directly from the camera. HREF is like the venable signal for HDMI. It is always high when there is valid data, when pixels are send. VSync is high after each frame. PixelClk is the camera s pixel clock, it has one rising edge during every pixel. resetn is the reset of this block. The data input consist of the data the camera sends out. While valid data is being send, the data during one clock cycles is the Y, U or V value of a pixel. The output video is equal to the RGB value of one pixel. The code of this block can be found in appendix A.2. Camera_decoder_0 HREF VSync PixelClk video[23:0] resetn data[7:0] Camera_decoder_v1_0 Figure 5.2: Camera Decoder A Finite State Machine (FSM) is used to determine what sort of data is coming in. When VSync is low and HREF is high valid data bits are coming in, so the state will change to the state where Y is read. The first byte is always the first Y value of the YUV422 format. After Y is read the state will change to the state where U is read. After that it reads the second Y value. The last state will read the V component. When this byte is read in, the FSM will go back to the state to read the first Y value. At this time, the previous read YUV values are converted to two RGB pixel values. This will go on until HREF becomes low or VSync becomes high. Then all will be reset to their initial values.

18 12 5. Implementation 5.3. Memory Interface The amount of block storage available on the FPGA isn t sufficient to buffer even one frame of data from the video input sources. Therefore an interface is generated to the DDR3 memory found on the FPGA board. The Vivado suite has a tool called the Memory Interface Generator (MIG), which was used to create a memory controller block. This block is shown in Figure 5.3. mig_7series_0 reset sys_clock S_AXI sys_rst clk_ref_i sys_clk_i aresetn DDR3 ui_clk_sync_rst ui_clk ui_addn_clk_0 mmcm_locked init_calib_complete ddr3_sdram Memory Interface Generator (MIG 7 Series) Figure 5.3: Memory Interface Generator The ports shown in Figure 5.3 can be explained as followed. The sys_rst signal resets the memory controller when the reset signal is low. The sys_clk_i signal is the system clock speed on which the controller should be running. This is not the same speed at which the memory is running, however. The MIG block internally generates a few different clocks using a Mixed- Mode Clock Manager (MMCM), on which the actual DDR3 memory is running. The protocol to efficiently use the DDR3 memory is complicated, so making use of the MIG interface allows for easily integrating of the memory with the rest of the system. The S_AXI port is the main data bus of the MIG. Using the AXI4-full protocol, data transfers are initiated by the bus master. The MIG is always configured as slave device, because is serves as a data endpoint. The AXI4-full protocol supports data bursts, which can be used to send blocks of data instead of single packets. This increases the throughput on the bus, because the DDR3 also supports burst transactions. The DDR3 port is a vector of different signals that interface with the actual DDR3 memory peripheral. The signal ui_clk is the output clock for the user interface components. These components could be connected using sys_clk_i, however the clock s origin is an external oscillator, which should be buffered before usage. The last important signal is init_calib_complete. After receiving a reset signal, the MIG executes certain sequences to reset and calibrate the DDR3 memory. The pin init_calib_complete will be asserted when the calibration process is complete, so it functions as a trigger for other components to start performing operations invoking the memory. More details on the Memory Interface Generator can be found in the documentation provided by Xilinx [21].

19 5.4. AXI-lite Reader AXI-lite Reader Connecting the developed components to the main memory can be done by adding an AXI interface port. As discussed in the previous section, most of the components need memory access and thus require a block that perform read operations. In Figure 5.4 shows the block design of the AXI-lite Reader component. AXI-lite was chosen in favor of AXI-full, because it is easier to understand and implement. Although the bandwidth of the AXI-lite interface is more limited than AXI-full, this won t be a problem as the bus frequency can be increased when more throughput is required. ram_reader_0 LK TN clk ble M_AXI_ACLK M_AXI_ARESETN ENABLE UI_ADDR[31:0] ram_reader_v1_0 M_AXI UI_RDATA[31:0] UI_RVALID DONE Figure 5.4: Block design of the AXI reader component etn In Figure 5.4 the following ports can be found: AXI, M_AXI_ACLK and M_AXI_ARESETN are part of the AXI-lite interface. UI_ADDR, UI_ADDR, UI_RDATA and UI_RVALID are signals meant to transfer data between the AXI reader and the component requesting data from the memory. The reader will start performing transactions when an address is written to UI_ADDR and the ENABLE pin is asserted. Output pin DONE is asserted when all transactions are executed. pixe rese clk_ hsy ven pixe read pixe The amount of transactions performed by the AXI Reader can be configured before synthesizing the design in Xilinx Vivado. The component will start reading data from the given address and wait until it receives a response from the AXI slave it is connected to. Next, the internal address offset counter is increased by the amount of bytes in the data bus (which is four in the case of Figure 5.4) and the next transaction is initiated. When the predefined number of transactions is reached, the reader stops requesting data and asserts output pin DONE. The user component connected to the AXI readers has to implement a system for sampling data from output pin UI_RDATA at the moment UI_RVALID is asserted. In a future version of this component an additional input pin will be created to notify the reader that the data on pin UI_RDATA has been received. This is useful for transferring data between two different clock domains: the clock domain of the AXI bus and the domain of the user component connected to the AXI reader. At the moment the reader only supports synchronous transfers.

20 14 5. Implementation 5.5. System Interconnects After explaining the implementation of the blocks themselves, the system overview is presented. Figure 5.5 shows the general overview of how the different blocks will be connected. The Sync Generator is part of the Video out Controller. Additional logic is included for buffering a row of pixel data. This is necessary because the data read from the memory returns in small packets of one pixel. It is not a continuous stream of bytes. Figure 5.5: Block design overview for the FPGA The FPGA input and output ports in Figure 5.5 are connected as followed: The HDMI input and the USB 2.0 output port are connected to the computer, the HDMI output is connected to the Oculus Rift and each camera is connected to its own camera input port. The center of the design consists of the AXI interconnect. Is has three slave ports and one master port for communication with the main memory. The interconnect is made using an IP block from Xilinx, which can automatically adapt between different clock domains and AXI implementations. There exist three types of AXI implementations: AXI-full, AXI-lite and AXI-stream. AXI-lite is a simple implementation of AXI-full, focused on a small amount of single packet transfers. AXI-stream is designed as a burst-only protocol for transferring huge amounts of data without using addresses. AXI-full is the preferred implementation, however it is the most difficult to create such a controller. At the moment the AXI-full type bus is only implemented in the connection between the interconnect and the memory controller. For more information about the AXI protocols, see the documentation provided by ARM [22].

21 6 Validation 6.1. Sync Generator The Sync Generator module can be tested by generating a clock signal and applying it to the module. The resulting image should show reasonable venable and hsync signals. This means that those two signals are not allowed to overlap. Also, when the reset and clk_enabled are held low, the synchronization counters should be reset to zero and the signals venable and hsync should not be asserted. Figure 6.1: Testing the horizontal synchronization signal In Figure 6.1 all expected values for the previously mentioned signals can be seen. Only when reset (number 1) and clk_enabled (2) both are asserted, output is seen at the venable (3) and hsync (4). As for the vertical synchronization, it is expected to occur after an predefined set of horizontal synchronization signals and have a duration of at least four hsync pulses. Figure 6.2 shows the expected result for the vsync (3) signal. Also, note the duration of the hsync (2) pulses in comparison to the previous figure. Figure 6.2: Testing the vertical synchronization signal 15

22 16 6. Validation 6.2. Camera Decoder The camera decoder is tested by generating a clock signal, a HREF signal, a VSync signal and data equal to known YUV sequences to check the computed RGB value. Figure 6.3: Testing the camera decoder In the simulation (Figure 6.3 it it shown that the decoder works. When VSYnc = 0 and HREF becomes 1, the incoming data is saved on every rising edge of the pixel clock. The data in red are the first four valid data bytes, Y1, U, Y2 and V. When this sequence has been received the conversion starts and while the next sequence is coming in the converted pixels are send out. The first two pixel are both white, because the sequence ff,80,ff,80 has been chosen (all these values are hexadecimal). This thus translates to ffffff in RGB format. The next four bytes are 01 for Y1, 04 for U, 63 for Y2 and ff for V. This is dark red. The converter converts these values to b30000, which means 179 Red (on the scale 0 to 255), 0 green and 0 blue. Thus also dark red. Because in this pattern Y1 and Y2 are different, a new RGB value is calculated after two clock cycles, instead of four in the first part, where Y1 and Y2 were the same.

23 6.2. Camera Decoder AXI-lite Reader One of the created blocks implements an AXI-lite reader to fetch blocks of data from the main memory using an AXI bus. The testbench was constructed as followed: first an IP package provided by Xilinx would write a series of known values into the block memory at a predefined address. When the writer block is done, it asserts an output flag. This flag is then used as enable flag for the reader block. Figure 6.4 was extracted from one of the simulation results. The yellow number 1 found in Figure 6.4 shows the point where the enable flag is being asserted, marking the point where the data was written to the block memory. At point 2, the AXI reader writes the requested address to output signal M_AXI_ARADDR and asserts M_AXI_ARVALID. Next, at number 3, the response from the AXI slave is received. In this case, M_AXI_ARVALID is set to a value of 1, which means the requested address is valid and the data will be retrieved from the memory soon. A few clock cycles later, the memory controller has performed a lookup on the requested address and returns the data stored there. At number 4 in Figure 6.4 can be seen that the signal M_AXI_RVALID is asserted. The response tells the AXI master that the lookup has succeeded and the data can be read from the data channel. After the AXI master has sampled the data vector, it sets pin M_AXI_RREADY to 1 to indicate the data has been read and the slave can continue processing other requests, as can be seen in the figure at number 5. In this case master directly issues another command, at number 6, but it might be possible that the slave will not allow starting a new request before other requests are completed. In case an error occurs during the request, instead of asserting pin M_AXI_RVALID, input signal M_AXI_RRESP will be set to a value different from 00. Because the signal consists of two bits and one combination is used to indicate there was no error, three types of error messages can be communicated. 01 is set when an operation having exclusive access has succeeded, 10 means an error flag was set at the slave device and 11 means the requested address does not point to a valid slave device. In the AXI reader design these errors are ignored. Figure 6.4: Results of testing the AXI-lite reader

24 7 Discussion of Results The subsystems discussed in the previous sections all work properly as independent modules as is demonstrated in Chapter 6, the validation. However, all the components still need to be connected with each other. This will not be very difficult, but can only been done when everything is working properly (when we can send the camera feed to the HDMI). The HDMI video signal can be received and an HDMI video signal can be generated. The interface with the camera also succeeds in translating the YUV color space into RGB values required by the modules programmed into FPGA. The biggest problem in getting the system to work are currently the cameras. The interface used to communicate with the camera is working pretty well, however the correct setting for the camera to get it to output 60 frames per second at a resolution of 720p with interpolated RGB values just does not seems to work. The camera has a lot of configuration options and the set of parameters that meet the requirements have not been found yet. 18

25 8 Conclusion and Recommendations 8.1. Conclusion At the time of writing there is still no working prototype. Therefore cannot be concluded that the concept is going to work. A working prototype is expected soon, but with a lower frame-rate than preferred. However, to show that the concept works, the highest possible frame-rate is not required. We expect that the concept will work, so further development on this project is recommended Recommendations Recommended for future projects is to find better cameras that are easier to implement in the total design. The FPGA chosen for this project met the requirements. The additional GPIO pins required for connecting the cameras were provided by the expansion board. The HDMI protocol is also recommended for use in future projects concerning the transportation of video streams, because of the good video quality and high reliability of the protocol. 19

26 A VHDL Source Code A.1. Sync Generator 1 library IEEE; 2 use IEEE.STD_LOGIC_1164.ALL; 3 use IEEE.NUMERIC_STD.ALL; 4 5 entity sync_generator is 6 Port ( clk : in std_logic; 7 resetn : in std_logic; 8 sw : in std_logic_vector(7 downto 0); 9 RGB : in std_logic_vector(23 downto 0); 10 btn : in std_logic; 11 vsync : out std_logic; 12 hsync : out std_logic; 13 venable : out std_logic; 14 displayout : out std_logic_vector(23 downto 0) ); 15 end sync_generator; architecture Behavioral of sync_generator is signal counterx, countery : UNSIGNED (10 downto 0); 21 --counterx is used for HSync, countery for VSync 22 signal H_1,H_2,H_3,H_4,V_1,V_2,V_3,V_4 : UNSIGNED (11 downto 0) := (others => 0 ); 23 --Resolution and frequency specific values for the HDMI modeline begin process (clk, resetn, counterx, countery) 28 begin Reset is active low! 31 if (resetn = 0 ) then 32 vsync <= 0 ; 33 hsync <= 0 ; 34 venable <= 0 ; 20

27 A.1. Sync Generator counterx <= (others => 0 ); 36 countery <= (others => 0 ); elsif (rising_edge(clk)) then 40 if (counterx >= H_2) and (counterx < H_3) then 41 hsync <= 1 ; 42 else 43 hsync <= 0 ; 44 end if; if (countery >= V_2) and (countery < V_3) then 47 vsync <= 1 ; 48 else 49 vsync <= 0 ; 50 end if; if ((counterx < H_1) and (countery < V_1)) then 53 venable <= 1 ; 54 else 55 venable <= 0 ; 56 end if; if (counterx = (H_4-1)) then 59 counterx <= (others => 0 ); 60 if (countery = (V_4-1)) then 61 countery <= (others => 0 ); 62 else 63 countery <= countery + 1; 64 end if; 65 else 66 counterx <= counterx + 1; 67 end if; end if; end process; process(btn) 74 begin 75 if (btn = 0 ) then 76 displayout <= ; 78 else 79 displayout <= 80 sw & sw & sw; 81 end if; 82 end process; INPUT MODELINE VALUES: 84 H_1 <= to_unsigned(1280,0); 85 H_2 <= to_unsigned(1360,0); 86 H_3 <= to_unsigned(1496,0); 87 H_4 <= to_unsigned(1712,0); 88 V_1 <= to_unsigned(1024,0); 89 V_2 <= to_unsigned(1028,0); 90 V_3 <= to_unsigned(1032,0);

28 22 A. VHDL Source Code 91 V_4 <= to_unsigned(1063,0); 92 end Behavioral; A.2. Camera Decoder 1 library IEEE; 2 use IEEE.STD_LOGIC_1164.ALL; 3 use IEEE.NUMERIC_STD.ALL; 4 5 entity Camera_decoder is 6 Port ( HREF : in STD_LOGIC; 7 VSync : in STD_LOGIC; 8 PixelClk : in STD_LOGIC; 9 resetn : in STD_LOGIC; 10 data : in STD_LOGIC_VECTOR (7 downto 0); 11 video : out STD_LOGIC_VECTOR (23 downto 0)); 12 end Camera_decoder; architecture Behavioral of Camera_decoder is signal y1, u1, y2, v1, y3, u2, y4, v2: std_logic_vector(7 downto 0); 17 signal y1_en, y2_en, u_en, v_en, leo, rw, pix, res: std_logic; 18 type state_type IS (Idle, write_y1, write_y2, write_u, write_v); 19 signal state, n_state : state_type; begin process(pixelclk, resetn) 24 begin 25 if (resetn = 0 ) then 26 state <= Idle; 27 res <= 1 ; 28 elsif rising_edge(pixelclk) then 29 if (res = 1 ) then 30 res <= 0 ; 31 state <= Idle; 32 else 33 state <= n_state; 34 end if; 35 end if; 36 end process; process(state, HREF, VSync, resetn, PixelClk) 39 begin 40 case state is 41 when Idle => 42 if (VSync = 0 ) and (HREF = 1 ) then 43 n_state <= write_y1; 44 else 45 n_state <= Idle; 46 end if; 47 y1_en <= 0 ; 48 u_en <= 0 ; 49 y2_en <= 0 ; 50 v_en <= 0 ;

29 A.2. Camera Decoder rw <= 0 ; 52 when write_y1 => 53 y1_en <= 1 ; 54 u_en <= 0 ; 55 y2_en <= 0 ; 56 v_en <= 0 ; 57 rw <= 1 ; 58 n_state <= write_u; 59 when write_u => 60 y1_en <= 0 ; 61 u_en <= 1 ; 62 y2_en <= 0 ; 63 v_en <= 0 ; 64 rw <= 0 ; 65 n_state <= write_y2; 66 when write_y2 => 67 y1_en <= 0 ; 68 u_en <= 0 ; 69 y2_en <= 1 ; 70 v_en <= 0 ; 71 rw <= 1 ; 72 n_state <= write_v; 73 when write_v => 74 y1_en <= 0 ; 75 u_en <= 0 ; 76 y2_en <= 0 ; 77 v_en <= 1 ; 78 rw <= 0 ; 79 n_state <= write_y1; 80 when others => 81 y1_en <= 0 ; 82 u_en <= 0 ; 83 y2_en <= 0 ; 84 v_en <= 0 ; 85 rw <= 0 ; 86 n_state <= Idle; 87 end case; 88 end process; process(pixelclk, resetn) 91 variable r_t, g_t, b_t, c_v1, c_v2, d_v, e_v, c_v3, c_v4, d_v1, e_v1 : integer := 0; 92 begin 93 if (resetn = 0 ) then 94 y1 <= (others => 0 ); 95 u1 <= (others => 0 ); 96 y2 <= (others => 0 ); 97 v1 <= (others => 0 ); 98 y3 <= (others => 0 ); 99 u2 <= (others => 0 ); 100 y4 <= (others => 0 ); 101 v2 <= (others => 0 ); 102 leo <= 0 ; 103 pix <= 0 ; 104 elsif rising_edge(pixelclk) then 105 if (y1_en = 1 ) then

30 24 A. VHDL Source Code 106 if (leo = 0 ) then 107 y1 <= data; 108 else 109 y3 <= data; 110 end if; 111 elsif (u_en = 1 ) then 112 if (leo = 0 ) then 113 u1 <= data; 114 else 115 u2 <= data; 116 end if; 117 elsif (y2_en = 1 ) then 118 if (leo = 0 ) then 119 y2 <= data; 120 else 121 y4 <= data; 122 end if; 123 elsif (v_en = 1 ) then 124 if (leo = 0 ) then 125 v1 <= data; 126 leo <= 1 ; 127 else 128 v2 <= data; 129 leo <= 0 ; 130 end if; 131 end if; 132 if (rw = 1 ) then 133 if (leo = 1 ) then 134 c_v1 := to_integer(unsigned(y1)); 135 c_v2 := to_integer(unsigned(y2)); 136 d_v := to_integer(unsigned(u1))-128; 137 e_v := to_integer(unsigned(v1))-128; 138 if (pix = 0 ) then 139 pix <= 1 ; 140 r_t := ((10000 * c_v1)+14075*(e_v))/10000; 141 g_t := ((10000 * c_v1)-(3455*d_v)-(7169*e_v))/10000; 142 b_t := ((10000 * c_v1)+17790*(d_v))/10000; 143 if (r_t < 0) then 144 r_t := 0; 145 elsif (r_t > 255) then 146 r_t := 255; 147 end if; 148 if (g_t < 0) then 149 g_t := 0; 150 elsif (g_t > 255) then 151 g_t := 255; 152 end if; 153 if (b_t < 0) then 154 b_t := 0; 155 elsif (b_t > 255) then 156 b_t := 255; 157 end if; 158 video <= std_logic_vector(to_unsigned(r_t,8)) & std_logic_vector(to_unsigned(b_t,8)) & std_logic_vector(to_unsigned(g_t,8)); 159 else

31 A.2. Camera Decoder pix <= 0 ; 161 r_t := ((10000 * c_v2)+14075*(e_v))/10000; 162 g_t := ((10000 * c_v2)-3455*d_v-(7169*e_v))/10000; 163 b_t := ((10000 * c_v2)+17790*(d_v))/10000; 164 if (r_t < 0) then 165 r_t := 0; 166 elsif (r_t > 255) then 167 r_t := 255; 168 end if; 169 if (g_t < 0) then 170 g_t := 0; 171 elsif (g_t > 255) then 172 g_t := 255; 173 end if; 174 if (b_t < 0) then 175 b_t := 0; 176 elsif (b_t > 255) then 177 b_t := 255; 178 end if; 179 video <= std_logic_vector(to_unsigned(r_t,8)) & 180 end if; 181 else std_logic_vector(to_unsigned(b_t,8)) & std_logic_vector(to_unsigned(g_t,8)); 182 c_v3 := to_integer(unsigned(y3)); 183 c_v4 := to_integer(unsigned(y4)); 184 d_v1 := to_integer(unsigned(u2))-128; 185 e_v1 := to_integer(unsigned(v2))-128; 186 if (pix = 0 ) then 187 pix <= 1 ; 188 r_t := ((10000 * c_v3)+14075*(e_v1))/10000; 189 g_t := ((10000 * c_v3)-3455*d_v1-(7169*e_v1))/10000; 190 b_t := ((10000 * c_v3)+17790*(d_v1))/10000; 191 if (r_t < 0) then 192 r_t := 0; 193 elsif (r_t > 255) then 194 r_t := 255; 195 end if; 196 if (g_t < 0) then 197 g_t := 0; 198 elsif (g_t > 255) then 199 g_t := 255; 200 end if; 201 if (b_t < 0) then 202 b_t := 0; 203 elsif (b_t > 255) then 204 b_t := 255; 205 end if; 206 video <= std_logic_vector(to_unsigned(r_t,8)) & 207 else 208 pix <= 0 ; std_logic_vector(to_unsigned(b_t,8)) & std_logic_vector(to_unsigned(g_t,8)); 209 r_t := ((10000 * c_v4)+14075*(e_v1))/10000; 210 g_t := ((10000 * c_v4)-3455*d_v1-(7169*e_v1))/10000; 211 b_t := ((10000 * c_v4)+17790*(d_v1))/10000;

32 26 A. VHDL Source Code 212 if (r_t < 0) then 213 r_t := 0; 214 elsif (r_t > 255) then 215 r_t := 255; 216 end if; 217 if (g_t < 0) then 218 g_t := 0; 219 elsif (g_t > 255) then 220 g_t := 255; 221 end if; 222 if (b_t < 0) then 223 b_t := 0; 224 elsif (b_t > 255) then 225 b_t := 255; 226 end if; 227 video <= std_logic_vector(to_unsigned(r_t,8)) & 228 end if; 229 end if; 230 end if; 231 end if; end process; end Behavioral; std_logic_vector(to_unsigned(b_t,8)) & std_logic_vector(to_unsigned(g_t,8)); A.3. AXI-lite Reader 1 library ieee; 2 use ieee.std_logic_1164.all; 3 use ieee.numeric_std.all; 4 use ieee.std_logic_misc.all; 5 6 entity ram_reader is 7 generic( 8 C_M_AXI_ADDR_WIDTH : integer := 32; 9 C_M_AXI_DATA_WIDTH : integer := 32; 10 C_NUM_READS : integer := ); 12 port( System Signals 14 M_AXI_ACLK : in std_logic; 15 M_AXI_ARESETN : in std_logic; Master Interface Write Address 18 M_AXI_AWADDR : out std_logic_vector(c_m_axi_addr_width-1 downto 0); 19 M_AXI_AWPROT : out std_logic_vector(3-1 downto 0); 20 M_AXI_AWVALID : out std_logic; 21 M_AXI_AWREADY : in std_logic; Master Interface Write Data 24 M_AXI_WDATA : out std_logic_vector(c_m_axi_data_width-1 downto 0); 25 M_AXI_WSTRB : out std_logic_vector(c_m_axi_data_width/8-1 downto 0); 26 M_AXI_WVALID : out std_logic;

33 A.3. AXI-lite Reader M_AXI_WREADY : in std_logic; Master Interface Write Response 30 M_AXI_BRESP : in std_logic_vector(2-1 downto 0); 31 M_AXI_BVALID : in std_logic; 32 M_AXI_BREADY : out std_logic; Master Interface Read Address 35 M_AXI_ARADDR : out std_logic_vector(c_m_axi_addr_width-1 downto 0); 36 M_AXI_ARPROT : out std_logic_vector(3-1 downto 0); 37 M_AXI_ARVALID : out std_logic; 38 M_AXI_ARREADY : in std_logic; Master Interface Read Data 41 M_AXI_RDATA : in std_logic_vector(c_m_axi_data_width-1 downto 0); 42 M_AXI_RRESP : in std_logic_vector(2-1 downto 0); 43 M_AXI_RVALID : in std_logic; 44 M_AXI_RREADY : out std_logic; User interface 47 ENABLE : in std_logic; 48 UI_ADDR : in std_logic_vector(c_m_axi_addr_width-1 downto 0); 49 UI_RDATA : out std_logic_vector (C_M_AXI_DATA_WIDTH-1 downto 0); 50 UI_RVALID : out std_logic; 51 DONE : out std_logic ); end ram_reader; Architecture architecture implementation of ram_reader is type state_type is (S_IDLE, S_ADDR, S_WAIT, S_READ, S_DONE); 63 signal current_state, next_state : state_type; signal offset : unsigned(31 downto 0); 66 signal axi_rready : std_logic; begin Master Interface Write Address 71 M_AXI_AWADDR <= X ; 72 M_AXI_AWPROT <= 000 ; 73 M_AXI_AWVALID <= 0 ; Master Interface Write Data 76 M_AXI_WDATA <= X ; 77 M_AXI_WSTRB <= 1111 ; 78 M_AXI_WVALID <= 0 ; Master Interface Write Response 81 M_AXI_BREADY <= 0 ; 82

34 28 A. VHDL Source Code Master Interface Read Address 84 M_AXI_ARPROT <= 001 ; Master Interface Read Data 87 M_AXI_RREADY <= axi_rready; State machine control 90 process(m_axi_aclk, M_AXI_ARESETN) 91 begin if (M_AXI_ARESETN = 0 ) then 94 current_state <= S_IDLE; 95 elsif (rising_edge(m_axi_aclk)) then 96 current_state <= next_state; 97 end if; end process; State machine process 102 process (current_state, offset, ENABLE, UI_ADDR, M_AXI_ARREADY, M_AXI_RVALID, M_AXI_RDATA) 103 begin 104 case current_state is when S_IDLE => M_AXI_ARADDR <= X ; 109 M_AXI_ARVALID <= 0 ; 110 axi_rready <= 0 ; 111 DONE <= 0 ; if (ENABLE = 1 ) then 114 next_state <= S_ADDR; 115 else 116 next_state <= S_IDLE; 117 end if; when S_ADDR => Set addr we want to read from 122 M_AXI_ARADDR <= std_logic_vector(unsigned(ui_addr) + offset); Set address valid bit 124 M_AXI_ARVALID <= 1 ; Default output values 127 axi_rready <= 0 ; 128 DONE <= 0 ; if AXI slave has accepted our address 131 if (M_AXI_ARREADY = 1 ) then 132 next_state <= S_WAIT; 133 else 134 next_state <= S_ADDR; 135 end if; when S_WAIT =>