Video Painting Group Report

Transcription

1 Video Painting Group Report Opal Densmore Kei-Ming Kwong Wahid Rahman Digital System Design (ECE532H1S) Prof. Paul Chow TA: Jasmina Vasiljevic April 10, 2014

2 Contents List of Figures... ii List of Tables... iii Chapter 1 Overview Background and Motivation Prior Work Why Use FPGAs? Project Definition & Goals System Overview System Description Key Definitions System Block Diagram Brief Description of IP... 5 Chapter 2 Outcome Summary Area of Improvements Lessons for Next Time Future Work Chapter 3 Project Schedule Chapter 4 Description of Blocks Significant IP Cores Other IP Cores Chapter 5 Description of Design Tree Chapter 6 Conclusion Tips and Tricks Video Demo References i

3 List of Figures Figure 1 Final system block diagram. The green blocks indicate existing IP blocks, while the red blocks indicate custom IP that was created as part of this project Figure 2 XPS System Assembly View of project Figure 3 The vmodcam IP core with modifications (red) for XY coordinate and Track Wand Figure 4 Track Wand IP logic Figure 5 vidcoord_xfer IP core Figure 6 The original HDMI_OUT IP core provided at the start of project [8] Figure 7 Modifications (red) made to HDMI_OUT IP core to integrate frame merging ii

4 List of Tables Table 1 - Acceptance Criteria for Project and Implementation in Final Project... 8 Table 2 - Proposed and Actual Project Schedule iii

5 Chapter 1 Overview 1.1 Background and Motivation The use of video processing in human-computer interaction (HCI) is expanding in applications ranging from entertainment, health-care, and education. Gaming consoles like the Xbox and its Kinect camera sensor augment users multimedia experience: computer vision and image processing algorithms built around the Kinect has been used to control ingame avatars based on user facial expressions and biometrics [1]. A biomedical company, Gestsure, is developing a Kinect-based system that uses surgeons hand motions to retrieve patient scans and images during operation [2]. In addition, studies note that the customized software to facilitate classroom interactions and to create Kinect-enabled contents seems to be missing in the picture of current technology integration [3] Prior Work While several video-based HCI systems have been developed as discussed previously, two systems implement a video painting system similar to this project. Virtual Whiteboard [4] This system develops an electronic chalkboard for teaching, without using specialized hardware. Instead, it uses generic camera and projectors. This software system uses Java and OpenCV libraries to detect and predict hand motions in real-time video. This system is a robust and feature-rich solution: it recognizes hand gestures by both hands at a real-time video frame rate (30 frames/sec at 320x240 resolution). However, the system only produces a screen of drawn shapes, not real-time video. cvpaint - Simple drawing using camera [5] This system is an OpenCV project that captures video from a webcam, tracks a user pointer (e.g. yellow block), and paint on the output streaming video based on location of pointer. This 1

6 project serves both as an inspiration and an example for this project. It is a simple algorithm implemented using the high-level OpenCV API. One downside is it relies on an OpenCVcompatible operating system. The reliance on software means limits the system s frame rate Why Use FPGAs? The benefits of using FPGAs for this project are twofold: hardware acceleration and reconfigurability. By implementing this project on an FPGA, custom hardware cores can be developed to accelerate video processing. This can directly improve frame rate. Reconfigurability allows for fast design cycles: redefining and testing software and hardware can be done relatively quickly using an FPGA. 1.2 Project Definition & Goals The goal of this project is to expand this video-based HCI to a proof-of-concept design on an FPGA. In particular, this project implements a video painting application that runs on video processing hardware in the FPGA. The motivation is to demonstrate real-time video HCI on an embedded FPGA system. In brief, the high-level goal of this project is to develop an embedded system on an FPGA that will output a video stream overlaid with trace of the arc a user draws in the air with a light source. The following are the specific goals of this project: Stream video input from a camera through the FPGA to video output on a monitor. Track the location of the light source paintbrush in the video input and store its coordinates within the video frame. Create a Draw Frame that is updated as the light source is tracked. For each tracked location of the light, a filled in circle (brush) is drawn at those coordinates in the draw frame. Merge the Draw and Video Frames into a Composite Frame. Output Composite Frame to HDMI Out. 2

7 Additional features of our project that were originally planned: Different drawing modes (colours, eraser, changing brush width) controlled by switches on the Digilent Atlys TM Board [6]. Clear screen functionality that resets the Draw Frame to an empty frame. Some applications of this project are as listed below: Interactive video, for a growing field for entertainment uses like the Kinect camera. Teaching, particularly for presentations conveyed over the Internet. Art, where our video drawing system could form a new creative medium. 1.3 System Overview System Description The design will make use of a Xilinx MicroBlaze soft processor to implement the Video- Painting algorithm. In order to implement a real-time system, critical software functions will be hardware accelerated. The system will integrate these hardware IP with the MicroBlaze to process real-time Video-Painting Key Definitions Draw Frame: An intermediate frame which is used to store the user drawn data Video Frame: An input frame from the camera which will be used as a background Composite Frame: An output frame with the Draw Frame overlaid on top of the Video Frame Paint Wand: The light source used for drawing. The location of the Paint Wand is tracked by Track Wand. 3

8 1.3.3 System Block Diagram DDR2 Draw Frame Video Frame Memory Controller VmodCam Video In VidCoordXfer HDMI Out Video Out Switches Track Wand Slave Registers Frame Merge AXI Bus MicroBlaze FPGA Update Draw Frame Update Brush Settings Figure 1 Final system block diagram. The green blocks indicate existing IP blocks, while the red blocks indicate custom IP that was created as part of this project. The system block diagram Figure 1 highlights the design and integration of various parts of our project. A camera records a video stream of a user. This video stream is captured by a Video-In IP block which controls the camera and stores the video stream on DDR3 memory. This is known as the Video Frame. When the user shines a light, the location of the light within the frame is tracked by a hardware IP, Track Wand. This hardware IP saves the location of the light in slave registers which are then read by software modules on the MicroBlaze. For example, a software module Update Draw Frame uses the tracked coordinates and draws a brush stroke (coloured square) at the location of the tracked coordinates in a Draw Frame (located on DDR3 memory). Next, a hardware IP called Frame Merge reads the Draw Frame and the Video Frame from memory, overlays the Draw Frame onto the Video Frame, and outputs the result to HDMI where it can be viewed on a monitor. 4

9 1.4 Brief Description of IP The following describe the functional blocks which were developed and/or used for this project. IP blocks which we were provided with are marked accordingly. 1. Update Brush Settings (Software) The Update Brush Settings block takes user input from the switches on the Atlys board and translates them into brush settings. Valid settings include brush colour, brush width, brush type (erase/draw), and a clear Draw Frame setting. The value of the switches is written to hardware slave registers and the values are read in software to be processed by the Update Brush Settings function. Inputs: Switch values read from hardware slave registers. Outputs: Brush setting values. 2. Update Draw Frame (Software) The Update Draw Frame block uses brush location, brush settings computed by Update Brush Settings and Draw Frame address to update the Draw Frame. The brush settings will be used to determine a stencil of the brush, which will be written to the Draw Frame at the specific brush location. Inputs: Brush settings, Paint Wand coordinates, draw frame memory address. Outputs: Updated Draw Frame, written directly to DDR3 memory. 3. Track Wand (Hardware) The Track Wand block monitors input video frames, locates a light source (hereinafter referred to as Paint Wand ) and outputs its coordinates to be stored in slave registers connected to the AXI bus. Inputs: Stream of pixel values and their coordinates within the frame from video-in IP as they are written to memory. Outputs: The horizontal and vertical pixel positions of the Paint Wand in the frame. 5

10 4. Frame Merge (Hardware) The Frame Merge block reads both the Draw Frame and Video Frame from memory, produces the Composite Frame based on the Video and Draw Frames, and writes the composite frame to the Video Out interface. Inputs: The memory addresses corresponding to the Draw, Video, and Composite Frames in DDR3 memory. Outputs: Composite Frame, written directly to Video Out. 5. Video In (Hardware, provided) [7] The Video In IP takes pixel input from the camera and stores it in DDR3 memory as the Video Frame. Inputs: Video data from camera. Outputs: Video Frame stored in DDR3 memory. 6. Video Out (Hardware, provided) The Video Out IP takes the Composite Frame and outputs it through HDMI to a monitor. Inputs: Stream of pixels composing the Composite Frame. Outputs: Data formatted with VSYNC and HSYNC for HDMI protocol. 7. VidCoordXfer (Hardware) This block holds slave registers that are used to make values calculated in Vmodcam (i.e. from Switch Settings and Track Wand) available to the MicroBlaze. Slave registers inside Vmodcam could not be used for this function because of the unique master/slave interface design of the existing Video-In IP: the core could be either a master or a slave for one data bus output, but the video input dominated the use and did not allow the MicroBlaze to access its internal slave registers. Inputs: Paint Wand coordinates, switch values. Outputs: Paint Wand coordinates and switch values stored in read-only slave registers which are read by MicroBlaze. 6

11 8. Vmodcam (Hardware, provided [7] and modified) Vmodcam serves as a wrapper for Video In and Track Wand. The switches pass through Vmodcam for debugging purposes and are then routed to VidCoordXfer to be used for brush settings as well. Inputs: Camera data, switch values. Outputs: Video Frame (to DDR3 memory), Paint Wand coordinates (from Track Wand, output to VidCoordXfer), and switch values (to VidCoordXfer). 9. HDMI Out (Hardware, provided [8] and modified) HDMI Out serves as a wrapper for Video Out and Frame Merge. Inputs: Memory location of Video Frame and Draw Frame. Outputs: HDMI data sent to a monitor. 7

12 Chapter 2 Outcome 2.1 Summary Overall, we were able to achieve what we planned to accomplish with this project. The final result is comparable to our initial design goals. The following are the requirements and the matching functionality within our system. Table 1 - Acceptance Criteria for Project and Implementation in Final Project Acceptance Criteria Implemented Status Tracking of Paint Wand Accurate tracking of Paint Wand given the following conditions: Slow paint wand movement Frame with no bright white colours Completed Fast Merging of Draw Frame and Video Frame Quick changing of brush settings Real-time video (30 frames/sec) Frame Merge IP merges the two frames at same speed as the output. An artifact of the frame merge is introduced where thin lines are seen above and below the overlaid objects. Brush settings take into effect immediately after selection. All the following modes are supported: Erasing Mode Brush Width Colour Settings Clear All Implemented hardware blocks to ensure video processing from camera to HDMI monitor was transferred at least 30 frames/second. Completed Completed Completed There are many constraints introduced which is required for the system to behave as expected. Despite the new conditions, the final system achieves the requirements as originally planned summarized in Table 1, thus the basic proof-of-design has been fulfilled. 8

13 2.2 Area of Improvements There are two main areas that should be improved to strengthen the usability of this design. The first is that the track wand algorithm, although hardware accelerated from our originally proposed software tracking algorithm, is still too slow to draw quickly. The user is required to move the light slowly through the air to draw. We spent a significant amount of time porting our software tracking algorithm to hardware and did not have time to optimize to make it faster for our demonstration. The second issue is the the video stream flips the input from the camera. The original intent was that the user would face both the camera and output monitor while using the design and that their image would look as if they were looking into a mirror. However, the camera flips this input so the output video stream is flipped in the vertical axis compared to the desired functionality. This means that the user must draw everything backwards in the air, which is a very challenging task. 2.3 Lessons for Next Time If we were to start from the beginning, the first change to our actions would be that we would not have invested so much in the software at the beginning. We relied too heavily on the software version of the Track Wand IP and writing the hardware version took up a lot of our time. We also spent time at the beginning implementing software features such as drawing in colours. These efforts should also have been put towards fixing hardware modules at the beginning. Although it was necessary to develop a software version of Track Wand to test the algorithm, we should have started on the hardware version much earlier than we did. This would have also freed up our time enough to implement other hardware features such as flipping the output frame so it mirrors the user s movements. The reason we had a late start to the hardware Track Wand is that we had planned for it to be in software, and only in hardware if it was not fast enough. What we should have planned is to make an initial software version, but always assuming that it would have to be ported to hardware. As a result of our attitude towards the block, we tried other software modifications such as optimizing the software tracking algorithm and trying to draw lines between tracked locations. This effort was in effect wasted and should have been put towards porting it to hardware earlier on. 9

14 2.4 Future Work If someone were to take over this project, there are two recommended next steps. 1. Accelerating the hardware Track Wand algorithm so that the IP can track fast movement seamlessly. 2. Work on flipping the video frame. Only after these hardware advances have been made, should they begin adding software features such as menus, paint-by-numbers games, and other features we had originally planned. 10

15 Chapter 3 Project Schedule Table 2 describes the project milestones week-by-week which were proposed and actually executed throughout the course of the semester. Table 2 - Proposed and Actual Project Schedule Week Proposed Deliverable Actual Deliverable 1 Kei-Ming: Finish software Track Wand IP with static inputs. Kei-Ming: Developed Track Wand software algorithm. Tested with an ideal static frame to verify tracking thresholds and coordinates calculation in software. Performance problem of initial algorithm noted. Wahid: Development hardware core for Frame Merge IP for use with static frames. Wahid: Initial development of hardware core for Frame Merge IP. Bugs encountered with corrupted video output. Opal: Initial development software Update Draw Frame and Update Brush Settings IPs independently with static inputs. 2 Kei-Ming: Verify functionality of preexisting Video-In IP. Opal: Software IPs Update Draw Frame, Update Brush Settings completed with static inputs for Update Draw Frame and switch inputs for Update Brush Settings. Kei-Ming: Verified functionality of Video-In IP and integrated with basic Video-Out. Tested coarser tracking algorithm, accuracy issues noted. Wahid: Develop AXI bus interface Wahid: Completed hardware Frame 11

16 control for Frame Merge IP. Merge core. Can merge two dynamic frames animated by MicroBlaze. Some horizontal corrupted lines, to debug. Opal: Complete software Update Draw Frame and Update Brush Settings IPs independently with static inputs. 3 Kei-Ming: Continue development in Video-In IP and test functionality. Opal: Integrated software IPs Update Draw Frame and Update Brush Settings with Track Wand and Frame Merge. Can now overlay a draw frame constructed from brush co-ordinates inputted from the keyboard. Kei-Ming: Integrated Video-In, Video- Out, and hardware Frame-Merge block. Tested Software Track Wand IP with input video frames (Still using software tracking; algorithm is slow). Wahid: Develop software control interface of Frame Merge IP. Wahid: Debugged Frame Merge IP to try to remove lines. Some corrupted lines at frame edges. Ran several simulations and tests, but bug not resolved. However issue was not detrimental to overall frame merging. Opal: Integrate software IPs Track Wand, Update Draw Frame, Update Brush Settings, and the software version of the Frame Merge IP using static inputs for all IPs. Note Update Brush Settings should use dynamic inputs. Opal: Made attempts to hide latency of Track Wand software IP by creating an efficient line drawing algorithm which drew lines between two tracked points. Although algorithm was successful, this did not hide latency as tracking was still slow and leading to inaccurate, choppy line 12

17 drawing. Frame Merge line bug also made some lines not visible once merged with the video stream. 4 Kei-Ming: Finalize software interface for Video-In IP Wahid: Test Frame Merge IP with software control interface. Opal: Integrate software with Video- Out IP. 5 Team: Integrate Frame Merge IP for use with dynamic frames (e.g. input video frames). Integrate software IPs Track Wand and Update Draw Frame with dynamic frames (e.g. input video frames) 6 Team: Integrate final system with dynamically changing Draw Frame (e.g. tracks user input) Team: Begun working on hardware Track Wand IP by reading in frame data as the Video-In IP wrote it to memory. Figured out how to keep track of the current location being written as well as how the pixel data was written to memory. Created a shift-register to store 10 pixels of data at a time as they were being written to memory. Team: Further development of the hardware Track Wand IP. Created an intermediate IP to hold slave registers so tracked coordinates can be read by the MicroBlaze. Simulated functionality of tracking, however, still not tracking correctly. Kei-Ming & Wahid: Implemented higher resolution XY coordinates Counter. Made several attempts to fix Track Wand hardware IP by testing for synchronization issues between XY counter and Track Wand IP. Fixed bugs related to tracking the wrong colour. Opal: Rewrote Update Brush Settings so switch values were also read from slave 13

18 registers by the MicroBlaze. This was done because switches were used by Video-In and could not also be hooked up to a GPIO. We did not want to disable the switches in Video-In for debugging. 7 Team: Everything working as specified in this document. Team: Discovered software hack to solve final bug of hardware Track Wand. The bug was that the wand was tracked at an x coordinate that varied in accuracy based on a linear function of the y coordinate. Almost everything works as specified. In summary, the key difference between these schedules is that we underestimated the difficulty of hardware tasks and overestimated the difficulty of software tasks. Software tasks were completed very early with hardware taking up most of our time. Hardware design of Track Wand was delayed until later in the term when we were very busy with all our courses. This caused us to need three weeks to build the hardware Track Wand IP, which we struggled with for too long to accelerate using only software. 14

19 Chapter 4 Description of Blocks This section describes the IP blocks and software algorithms used in this project, their detailed functionality, and integration with the rest of the system. Figure 2 highlights the IP blocks used and bus connections at a high level. The IP blocks are discussed in more detail below. Figure 2 XPS System Assembly View of project. 4.1 Significant IP Cores The key IP cores unique to functionality of this project were vmodcam (with wand tracking), vidcoord_xfer, and hdmi_out (with frame merging) as discussed in this section. vmodcam v1.00.a This block was used as the Video-In core in the system architecture. In addition, to minimize memory bus usage, this core was also responsible for tracking the location of the 15

20 Paint Wand. The Video-In IP core was taken from another group responsible for working on the Video-In IP for the IP project [7]. The provided Video-In block has two cameras as input and outputs to DDR memory using the AXI bus interface. It is configured to work with RGB 565 or 16 bit wide pixels. The provided IP block works by writing each camera s data stream to their individual FIFO and bursting that onto the DDR when one of them reaches a specific threshold. In particular, it bursts out 32 pixels using the 32-bit AXI bus by sending two 16-bit pixels each transfer. Two key changes were made to the vmodcam IP core, as shown in Figure 3: A Track Wand IP block to determine if the window is considered matching An XY coordinate counter to determine the current XY location of the window in relation to the frame. This implementation is useful because it allows Track Wand IP to skip accessing the memory to read from the Video Frame. By intercepting the video stream, it reduces memory accesses. The following diagram shows the final architecture of the Video In IP with the additions: Figure 3 The vmodcam IP core with modifications (red) for XY coordinate and Track Wand. 16

21 The XY Coordinate Counter increments with the control signals from the camera control. During normal Video-In access, whenever a pixel is streamed into the FIFO, it causes certain control signals to be raised. Using these control signals, and the known reset conditions, it is possible to determine the XY location of the current pixel. The Track Wand block intercepts the video stream latches the XY coordinates of the first window which matches the thresholds. The architecture of the Track Wand IP is shown in Figure 4. Figure 4 Track Wand IP logic. The pixel stream is stored in a 10 pixel wide shift register each pixel is shifted in from the video stream when it is being sent to memory. From this, the Track Wand IP will process each pixel in parallel to determine if it matches the thresholds for each pixel. This core is called the Per Pixel Calculation Core. If each pixel matches a bright white colour, it is considered a matched pixel. From that, the number of pixels in the window that match the criteria will be summed up. This is then compared to another threshold value which will determine if the number of matching pixels is higher than required. This effectively averages the per pixel colour thresholds across all the pixels across the window and will only trigger when most of the pixels match the criteria. vidcoord_xfer v1.00.a This block was created in this project to transfer coordinates from the Track Wand logic in vmodcam IP to the MicroBlaze. This was necessary due to the unique design 17

22 of the existing vmodcam IP core. The IP core provided had one interface to an AXI bus, and could act as either a master or a slave on that bus. However, the master and slave states were mutually exclusive; a mux switched between master-style data and slave-style data on the IP to AXI data bus. Since vmodcam was constantly a master to the memory controller and writing incoming video data to memory, it could never switch into its slave role for reading by the MicroBlaze. To work around this, the new IP core vidcoord_xfer was created. This core was dedicated as a slave-only IP to the MicroBlaze through an AXI slave bus. As shown in Figure 5, the vidcoord_xfer core received directly from the vmodcam IP x-coordinate, y-coordinate, and an enable signal indicating a paint wand was found at that location. The location of this wand is saved in slave registers within the vidcoord_xfer IP, from which the MicroBlaze can read for further software processing. Figure 5 vidcoord_xfer IP core. hdmi_out v1.00.a This block was used for outputting the video frames to an HDMI screen. This block was reused from a previous ECE532 project [8] which was made available as an IP project early in the course. The function of this block was to read memory contents at a pre-determined location in memory. This data was read over an AXI memory-mapped bus. The data was taken as pixels in a frame, and the pcore stored this data in a line buffer, a FIFO of depth bit data words. The core then synchronized this data with transfer to an HDMI output 18

23 block for display on a monitor. Figure 6 shows the HDMI_OUT core that was provided at the start of the project. Figure 6 The original HDMI_OUT IP core provided at the start of project [8]. The main change that was made to this block was the addition of the Frame Merge logic, as shown in Figure 7. This new frame merging logic was integrated into the existing HDMI_OUT core. As part of this project, two changes were made: Addition of FIFO selection logic for targeting writes from the AXI bus. This allowed the core to alternate between reading single lines of the Video Frame and Draw Frame. Addition of frame merging logic, which determined if a given pixel of Draw Frame should replace that of Video Frame (the merging operation). The output of the merging operation, done in real-time with the streaming video data, was then sent to the HDMI output logic for display on a screen. 19

24 Figure 7 Modifications (red) made to HDMI_OUT IP core to integrate frame merging. 4.2 Other IP Cores axi_interconnect v.1.06.a The AXI memory-mapped interconnect in crossbar or shared-access architectures for connections between AXI masters and slaves [9]. The interconnect was clocked at 100 MHz. One instance of this interconnect (axi4_0) was used to interface multiple IP blocks with the memory controller (MCB_DDR2) in a crossbar architecture, while another instance (axi4lite_0) was in a shared-access architecture for the MicroBlaze to communicate with its slave IP blocks. lmb_v10 v.2.00.b Local memory bus (instantiated for data memory and instruction memory) for MicroBlaze [10]. microblaze v8.40.a MicroBlaze processor. Debugging capabilities and instruction/data caches (each 4 KB) were made available [11]. 20

25 bram_block v.1.00.a On-chip memory of size 64 kb for the MicroBlaze. lmb_bram_if_cntlr v.3.10.a Interface controller for local memory bus to BRAM for MicroBlaze. axi_s6_ddrx v.1.06.a Memory controller for external DDR2 memory. The memory controller block (MCB) was built with this top-level AXI to MCB bridge to connect to AXI busses [12]. The DDR2 memory ran at 333 MHz. One 32-bit wide bi-directional port on the MCB was used, which connected to the AXI memory-mapped bus for control by various video IP blocks. The address space was 128 MB. mdm v2.10.a Debug module for MicroBlaze. axi_uartlite v.1.02.a UART interface from AXI to computer terminal for printing debug messages. Baud Rate of 9600 and false parity were used. clock_generator v.4.03.a Generates system clocks except for HDMI output video clocks. 100 MHz external clock is brought in through a GCLK network as clkin. This IP generates 6 clocks: 2 clocks at 600 MHz at 0 and 180 phase shifts for the DDR2 memory controller; 2 clocks of 24 MHz fat 0 and 180 phase shifts for the vmodcam video input controller; a 25 MHz clock for the pll_module dedicated to video output clock generat; and a 100 MHz system clock for all other internal blocks. 21

26 pll_module v.2.00.a Generates the video output clocks. A 25 MHz clock is input from clock_generator. A 250 MHz, 50 MHz, and 25 MHz clocks are generated for internal use in the HDMI video synchronization logic. 22

27 Chapter 5 Description of Design Tree This section highlights the design tree of our project and describes key files and directories. Video_Painting - top level directory README system.xps - XPS project data - contains System Constraints File system.ucf doc - contains documentation for project Demo Presentation Final Report pcores - hardware IP blocks used in project hdmi_out_v1_00_a vidcoord_xfer_v1_00_a vmodcam_v1_00_a original_pcores - original pcores for hdmi_out and vmodcam workspace track_wand_0 src - Contains software source code. main.c - main program to run Update Draw Frame based on tracked values and switch inputs update_brush_settings.c, update_brush_setting.h - Update Brush Settings software IP update_draw_frame.c, update_draw_frame.h - Update Draw Frame software IP (Ignore other files from previous iterations) 23

28 Chapter 6 Conclusion For this project, we set out to create an interactive video painting using an embedded system on a Xilinx Spartan-6 FPGA. We wanted to show that real-time video HCI was possible using such a system. Our task was to design, implement, and demonstrate this system within the course of the semester. We were able to meet our initial project goals. We were able stream video input from a camera, process it through an FPGA, and output the video to a monitor. We were able to track the location of a paintbrush (a light source) in the input video and store its coordinates. We were then able to use these coordinates to create a frame that recorded and drew markers based on the tracked location. We were able to merge this Draw Frame with the video going out to the monitor: the end result was real-time video output overlaid with painting done by the user with the light source. We were also able to implement several extra features, such as different colours, erasers, screen clearing, and brush widths, which were controlled by the switches on the Atlys board. 6.1 Tips and Tricks We learned some valuable lessons in using the Xilinx tools and digital design. 1) When simulating blocks that have an AXI interface, it is often difficult and can complicate simulations by having all the AXI signals toggling. The AXI signals can be fully simulated (an AXI Bus Functional Model, or BFM, provided as an encrypted simulation block by Xilinx). However, as a very first step, sometimes it is easier to isolate the IP under test in a simulator (e.g. Modelsim), and force toggle (e.g. a DO script) the relevant AXI signals (read, write, data, clk) to see if your IP is storing, reading, and writing data correctly and at the right times. [13] shows the correct timing for important AXI signals for different read and write modes. This allows fast compile and testing of your initial Verilog code. A BFM can then be added for system-level simulation testing. 24

29 2) Always clean your XPS project before recompiling, when you make IP changes. This is particularly true for VHDL, where we noticed XPS did not detect small changes to our VHDL source code and so did not recompile. Cleaning the project from the XPS menu solved this problem. 6.2 Video Demo Here is the link to a video demonstration of our project: 25

30 References [1] Z. Zeng, Microsoft Kinect Sensor and Its Effect, IEEE Multimedia, vol. 19, no. 2, pp. 4-10, [2] M. Campbell, Kinect imaging lets surgeons keep their focus, New Scientist, vol. 214, no. 2865, p. 19, [3] H. J. Hsu, The Potential of Kinect in Education, International Journal of Information and Education Technology, vol. 1, no. 5, pp , [4] F. Tavakolizadeh. (2014, Jan. 10). cvpaint - Simple drawing using camera [Online]. Available: [5] M. Lech, B. Kostek, A. Czyzewski. Virtual Whiteboard: A gesture-controlled pen-free tool emulating school whiteboard, Journal of Intelligent Decision Technologies, vol. 6, no. 2, pp , [6] Digilent Inc. (2014, Apr. 9). Atlys TM Board Reference Manual [Online]. Available: [7] ECE Group 3 IP Project. VmodCAM_Team3.rar. Available: Piazza.com course website, post [8] ECE Group IP Project. hdmi_out. Available: [9] Xilinx. (2014, Apr. 9). AXI Reference Guide [Online]. Available: 26

31 [10] Xilinx. (2014, Apr. 9). Local Memory Bus (LMB) V10 (v.1.00a) [Online]. Available: [11] Xilinx. (2014, Apr. 9). MicroBlaze Processor Reference Guide [Online]. Available: [12] Xilinx. (2014, Apr. 9). Spartan-6 FPGA Memory Interface Solutions [Online]. Available: [13] Xilinx. (2014, Apr. 9). LogiCore IP AXI Master Burst [Online]. Available: ds844_axi_master_burst.pdf 27