ECE 532 Group Report: Virtual Boxing Game

Transcription

1 ECE 532 Group Report: Virtual Boxing Game Group 18 Professor: Paul Chow TA: Vincent Mirian Ryan Fernandes Martin Kovac Zhan Jun Liau

2 Table of Contents 1.0 Overview Motivation Goals and Description System Block Diagram Brief Description of IP's Diagram of Game Outcome Comparison of Original and Final Features Assessment of Operation Improvements for the System Future Work Possibilities Review of Project Schedule Description of Blocks Microblaze Game Code Video Switching Color Detection Setup Color Detection Block (Paddle Detector) Fast Simplex Link (FSL) Video to RAM Video Switch Plb_tft_cntrlr Xps_interrupt_controller Plb_uart Xps_timer Debug Module LED debug mux Clock Generator MPMC Description of Design Tree References

3 1.0 Overview An overview of the project motivation, details, goals, and system blocks. 1.1 Motivation In recent years there has been an increasing trend to create motion tracking and touch user interfaces to interact with computing devices as seen on the Nintendo Wii, Microsoft Kinect, and touch screen phones/tablets. As such, our team thought that it would be interesting to work with motion tracking and alternative user interfaces in a game environment for this course project. By using two cameras, one tracking an x coordinate from above, and one tracking a y coordinate from the side, a screening space can be defined that maps directly to a VGA monitor. Such an interface allows direct tracking of a user's motion that corresponds to a screen without having to physically touch the screen. 1.2 Goals and Description The purpose of this project is to create a virtual boxing game. This is done by allowing users to punch towards the VGA screen and translate the position of a fist into coordinates relative to the screen. Since one camera has no depth perception, two cameras are required to correctly identify the position of the fist. The punches are to be aimed at diamond targets that are displayed randomly on the VGA screen. The cameras will be placed facing down and to the side, which allows each camera to detect one set of coordinates; X for the top camera and Y for the side camera. Through switching the video input of the two cameras and writing their inputs into separate memory locations, video data is obtained for the screening area for both x-z and y-z coordinate planes. A video processing block obtains the x and y coordinates by inspecting each video frame in memory (one x frame, one y frame) for the colored fist that corresponds to a punch through a color detection algorithm. The Microblaze CPU will compare the video coordinates from the video processing block to the diamond target coordinates on screen, if the coordinates are reasonably close, then the target will be hit as indicated by a yellow marker. The Microblaze CPU draws the diamonds and game visuals in software. Targets will randomly appear on the screen at regular intervals, with the intervals decreasing as the user successfully hits more and more targets. A counter will be used to record the score of the player, level time, and also to record the calories burned and the reaction time of the user. Therefore, in order for the project to be a success, the camera detection must be done in real time, and the accuracy of the detection must be within at least a few centimeters. 3

4 1.3 System Block Diagram Figure 1: System Block Diagram 1.4 Brief Description of IP's A brief description of the IP's from the block diagram is listed below. Video to Ram: Uses a timer and control signals from the Microblaze processor to switch between the 2 camera inputs and stores them to DDR RAM. This block was taken from the Video to RAM code and modified to work with 2 cameras. Video Switch: Relays signals between the Microblaze s PLB bus and the Video to Ram block. This block was newly created. Paddle Detector: Reads the frames from DDR RAM memory and searches for a predefined color, once the color is found it sends the coordinates using an FSL to the Microblaze processor. This block was taken from "Move-o-Phone" project and modified to work with our project. 4

5 Microblaze: Runs our game software, that will be used to display targets to be punched by the user. Additionally controls the video switching and color to detect for the paddle detector. Software originated from Wild West Project. IIC: I2C communication block for video decoder communication. From Video to RAM code given. Plb_tft_cntrlr: VGA output core, to display graphics on screen. From Video to RAM code given. Xps_interrupt_controller: Manages multiple interrupts over PLB. xps_intc Ver 1.00a from the IP catalog Plb_uart: UART module for debugging data over RS232. From Video to RAM code given. Xps_timer: Produces an interrupt at a set time interval. xps_timer ver 1.00a from the IP catalog. Debug Module: Debugs Microblaze through XMD and JTAG. From Video to RAM code given. LED debug mux: Debug multiplexor output for LED debugging. From Video to RAM code given. Clock Generator: Generates outputt clocks from a given input clock though PLL and DCM. From Video to RAM code given. MPMC: Allows multiple devices to access memory through ports. From Video to RAM code given. DDR Memory: External RAM of 256MB. From Video to RAM code given. 1.5 Diagram of Game Figure 2. Diagram of the external components of the game 5

6 The game consists of two video cameras, one pointed down and one to the sides as can be seen in figure 2. The user punches towards the screen, and the two cameras are each responsible for one coordinate; X for the top camera and Y for the bottom camera. The red X on screen represents the location that is detected corresponding to the position of the fist relative to the screen. 2.0 Outcome An analysis of the final design, assessment of operation, challenges, improvements that we could make, and a future outlook. 2.1 Comparison of Original and Final Features All of the original features indicated in the proposal have been included in the final product in some shape or form. Changes from the original include having diamond shaped targets instead of creatures to hit and detecting a colored fist as opposed to a generic hand to better accommodate the color detection algorithm. The Microblaze uses an FSL to obtain coordinate data instead of reading from memory. Score, level time, response time, and calories burned are added features displayed on the VGA. Multiple levels are supported, with the diamond target moving faster at higher levels. Below are the final features with the changes mentioned above. The following are the features of the project: Diamond targets show up on the screen at random locations drawn by the Microblaze CPU The user punches toward the screen at the target s location, stopping before impact to score One camera on top and one on the side identifies the colored fist location A hardware block translates colored fist location to X and Y coordinates A software block compares fist location to identify a hit or miss of the target Score, level, level time, response time, and calories burned displayed on screen Multiple levels of increasing movement speed for diamond target 2.2 Assessment of Operation Listed below are the original functional requirements from the project proposal Functional requirements: Must be accurate down to a couple of centimeters Must do processing in real time Must use two cameras Hardware block will perform video processing to determine X and Y coordinates from input video stream Microblaze processor will process X and Y coordinate data and map it to the VGA screen to identify a hit or miss 6

7 The measured accuracy of the project is within around 5cm, which meets the first requirement. The processing is all done in real time, however there is a video input and output delay, but it is consistent and not growing through the game operation. There are two cameras used, one for x and one for y coordinate determination. A hardware block does process x and y coordinate data, averages it and sends it to the Microblaze. The Microblaze does take this data over the FSL and maps it to the screen to identify a hit or miss for the targets. A detailed assessment of each major module is given below. The Video to Ram and Video Switch cores work very well. The switching between cameras is flawless, however the switching time is somewhat longer than we expected. The color detection module was largely successful. The module was intended to detect a certain color and find the centre of a large block of color. It was able to do this successfully and also was able to process the data from the cameras in real time. The module was also able to push the coordinates into a Fast Simplex Link (FSL) which allows the Microblaze C code to collect the coordinate data quickly. The block was instantiated twice, once for each camera, and each block was able to process the correct video frame and find the centre of a large block of color. The Microblaze code is able to draw a diamond on the screen which is able to randomly move through the screen using the rand function. Through a timer, the set interval of redrawing the diamond is controlled explicitly. This timer interval is reduced as the level increases, making the diamond drawn more quickly. The FSL is read through the timer, when the video frame is ready and emptied out as quick as possible to reduce video output delay on the screen and to ensure that the latest coordinates are being read. However, the video input delay is around 1 second, and so hitting the target is not perfectly synced with the punch motion. Score, level, level time, response time, and calories burned appear on the screen based on the user's performance in the game. These features seem to work as expected. 2.3 Improvements for the System For the video switching, one way to improve the switching timing is to create interrupt signals in the video switch block and trigger an interrupt in the processor based on this signal. During the interrupt the processor will reconfigure the video decoder chip to switch the video input. Improvements could be made for the FSL to cause an interrupt in the Microblaze when a pair of coordinates are available. This would make it easy for software to read the data from the hardware and will let the software know when the data is available immediately. For the Microblaze code, better graphics could have been used for the game. A smarter averaging and prediction code could have been created to compensate for the video input delay, reducing the appearance of lag for hitting targets. 2.4 Future Work Possibilities As the next steps I would recommend reducing the input video delay and video processing lag. Currently the delay is around 1 second and ideally we would want the delay to be less than 100ms, to have a smooth and accurate game that works at high speeds. Improved video graphics would go a long way to improving the visual prowess of the game, including creatures instead of diamond targets to hit, and 7

8 visual effects to indicate fast punches and fast response times. An audio effect that is played when a target first appears, would improve the overall user experience and fun factor. Finally having specialized targets, that require a certain response time to be hit, or a certain punch speed to score a point would be added for extra challenge. One suggestion for future work is for the project to identify the speed of the punch. This allows the user to vary the strength of the punch and can add a new depth to the game by requiring different speeds for different targets. This will require the video switching to happen relatively quickly so that the coordinates can be stored and compared with the previous set of coordinates without noticeable delay. Another suggestion is to add in support for a second player, as the color detection module can already detect a second color. Some modifications are required to add in support for the output of the second pair of coordinates through the FSL, and for the block to detect the centre coordinates of the second color block. As well, some rethinking of the game is required to add additional targets or some other purpose for the second player. 3.0 Review of Project Schedule Below is a table comparing our proposed milestones with our actual progress throughout the semester. Throughout the development of our project our group was able to keep up with our weekly milestones fairly well until around March 21, where we had to delay all progress by around 1 week due to having a busy schedule and having to submit a group report for another course, ECE496. In the beginning we completed locating sample code and setting up the video cameras for switching between two inputs on time. Problems began when we attempted to modify the Video to RAM p-core to allow for storing and loading video in memory. Work was started on Feb 29, however due to numerous technical challenges was not completed until Mar 28, with an expected completion date originally of Mar 7. The challenges included difficulty in analyzing the video to RAM p-core to modify its FSM to store and load video data, and creating a second address to store video data. We felt as if we were working blind until we ran simulation using ModelSim as per the advice of the TA, and convinced ourselves that the video was indeed being stored in memory. The video processing algorithm came together much as initially expected under the original schedule. The Microblaze game code was delayed by around 1 week, due to the previously mentioned delay. It was originally assumed that the game code would be completed before integration, however this assumption turned out to be optimistic, as many of the bugs in software were not ironed out until after integration occurred, additionally new game data was output on screen with new input from the TA at the end of March. Overall our method of splitting the work into three parallel parts (video switching, video processing, game code), prevent us from having any major bottlenecks from a single part not being completed. This allowed us to have our game completed for a system test by the original April 4 deadline. 8

9 Date Proposed Milestone Actual Progress Feb 8 Find Sample Code Found the Video to RAM sample code to For the software and hardware blocks, sample provide our video input interface. code will be searched for that implements as Looked into the Laser Pointer many features as possible to display video from a video camera, process video through Game and Interactive Snake Game for a color detection module in Verilog. edge detect/color and display output on the Read through the datasheets of the video VGA. daughter board to identify how to setup 2 Feb 15 Setup Video Camera Perform the necessary software and hardware changes from the given video input code to input 2 video camera sources alternatively cameras switching. Modified the video setup of the Video to RAM software code to switch between multiple analog inputs using the I2C interface on the fly, allowing us to connect 2 composite inputs to our 2 video cameras Used a switch to demonstrate switching from camera 1 to camera 2 Feb 29 Store and Load Video in Memory Be able to store and load video in memory. Assess if video buffers are required for video storage. Feb 29 Detection Algorithm Determine and code an algorithm to identify that an object has entered the viewing area. This algorithm must be able to differentiate the background from unidentified objects. Stored a single frame in memory and displayed it on the VGA output by modifying the Video to Ram Verilog Implemented a color detection algorithm from the Move-o-Phone project. We were able to change the color detected from red, blue and green Mar 7 Mar 7 Microblaze Game Interface Create preliminary code to read x and y coordinate data from memory and display output on screen at an acceptable speed. Determine delay and bandwidth headroom Determine the delays, and the inherent bandwidth of the system with particular focus on the video storage and processing, with aid from the snoopy core. The video processing algorithm was created utilizing the color detection algorithm from the "paddle detector" core, and adding an averaging block to find the center point of the detected colored object. An FSL was added to output the coordinate data. Switching between video frames in memory was added. A counter records the number of lines written and switches between inputs at the end of a frame and switches the memory location to store the video data. A new IP core called video switch was added that takes 2 frame done signals and writes them to a software accessible register, where the Microblaze can switch frames by writing to these registers The game code was started and included a drawing diamond algorithm 9

10 Mar 14 Processing Algorithm Determine and code an algorithm to process the fist on the screening area and determine a relatively accurate position map that can relate the screening area to the VGA screen to be stored in memory. Mar 21 Implement Verilog Hardware in EDK Implement the Verilog hardware block into EDK for a more system oriented test environment. The paddle detector code was modified to include an FSM that checked for reading the whole frame before outputting new coordinate data Integration of the video processing Verilog and the game code was done. The game code recognizes the coordinate data from the FSL and matches it to the target coordinates of the drawn diamond, increasing score. The video switching Verilog was improved to fix timing related issues, and an overwrite issue with the video data Due to a busy work schedule during this week including having to submit our ECE 496 report, we moved the milestone one week forward. Mar 21 Software Game Visuals and Speed Control Implement advanced software visual effects such as creatures on the screen/ targets to hit. Also implement speed control to modify the speed at which targets appear and disappear after being hit. Mar 28 Verification of Video Processing Algorithm Verify that the video processing algorithm is providing adequate position data and optimize as necessary. Mar 28 Verification of Game Code Verify that the game code is stable, and will be able to time well with hardware demands. Apr 4 System Test Connect the hardware and software blocks together through EDK and test the hardware and software code/description. 10 The game code includes features of multiple levels of increasing speed, level time, and calorie count. A timer was added to allow for speed control of levels. Software averaging was performed to smooth out the coordinate data from the FSL Integration of the video switching and video processing Verilog was completed Two instances of the color detection algorithm were included for both cameras, working at two different video addresses Integration for the video switching, video processing and game code was completed Additional averaging of the coordinate data was done Calorie count, levels, score, response time are displayed on screen now Video lag reduced in software by reading the FSL faster, when each frame is done for each camera, but not too fast as to cause instability Table 1: Project Schedule Comparison

11 4.0 Description of Blocks A description of the major blocks used in the block diagram with detailed analysis of the IP. 4.1 Microblaze The below diagram represents the software flow of the project. High Level Software Mid Level Game Code Video Input Switching Color Detection setup Low Level Draw Diamond Read FSL + Compare Diaomd Location Update On Screen Game Data Switch Video Input when frame done Set color to pink Figure 3: Software Block Diagram The following software is detailed: game code is derived from the "Wild West Game" (2011), the control of the color detection is derived from the "Move-o-Phone" (2011), and the video switching is derived from the "Video to RAM" version 1.00a. A timer has been setup with a time slice of 20ms, to perform all game code functionality, and keep the while loop in main() empty to optimize performance Game Code The original code from the Wild West project was modified to work completely off the timer interrupt. The VGA drawing library and implementation is derived from Andrew Shorten (2009) and used as is. New IP of drawing diamonds and circles were created, as well as target detection code, and game data code. In the timer, a diamond is drawn in a random location using rand() after a counter has finished incrementing to the max value Timerset=150. Every time a level has been increased this max value is decreased by 6 and the diamond is redrawn in a new location sooner. The max value has a lower bound of 30, which is the max speed of the game. The new diamond location is checked to ensure it is greater than +/- 150 pixels from the original diamond to ensure double hits are avoided. When a frame_x_done signal (1 or 2) is asserted its respective FSL (1 or 2) is read for new coordinate data. This process is done in the timer which has a time slice of 20ms to read the FSL not too quickly but not too slowly, to get valid coordinate data, avoiding reading delayed points, or garbage data from a blank frame when the video input has been switched. The previous coordinate data and current coordinate data are averaged [coordxavg2=(c old +c new )/2], and this averaged coordinate data is compared with the current diamond location. Averaging is necessary to avoid erroneous points from 11

12 being drawn from FSL coordinate data, which occur occasionally. To make the game easier, there is a threshold of +/- 65 pixels from the center point of the diamond to guarantee a hit. Once hit a yellow happy face is drawn over the target. Game data is redrawn on the screen after a new diamond has been drawn to optimize the performance of the C code. Score is kept by detecting a punch within the diamond threshold from the FSL data. Level is increased every 2 points of score. Response time is calculated based on the time slice and time till a punch has been detected, it is updated once a hit occurs. Level time is calculated as the current time of the level, and updated every second. Calorie count is calculated based on the number of punches on the screen, which assumes that every time a set of coordinates detected beyond a threshold of +/- 100 pixels is a new punch. Through using an average body weight of 180lb and an online calorie count calculator for boxing which bases calories on boxing time, the calorie count is calculated assuming 90 punches per minute Video Switching The original code from the Video to RAM project was modified to include a function to enable the component green input as a composite input through reprogramming the I2C. In the timer, the frame_x_done (1 and 2) signals are read from software accessible registers as asserted by the Verilog from the Video Switching module. Once a signal is asserted as done, the software writes to the register indicating that it will be switching inputs through frame_x_start signals (1 and 2). Video switching is done through reprogramming the Video daughter card through I2C, to switch between composite inputs for camera 1 and Color Detection Setup In the main code, the color detection algorithm is initialized to detect for pink/purple colors, by writing directly to memory with the HEX values for the RGB low and high threshold. This is modified from the original red, blue and green colors from the "Move-o-Phone" project. These low and high thresholds determine the acceptable RGB values that can be taken as input data. 4.2 Color Detection Block (Paddle Detector) The algorithm for the color detection was largely borrowed from the 2011 project Move-o-Phone, which in turn borrows the code from the 2010 project Virtual Pong. The block in its original form detects the first and last points of a certain color, and writes the coordinates to RAM for the Microblaze to collect. However, some modifications were made to this block for our project. The color detection block takes the system clock, system reset, frame_done and MPMC_DoneInit signals as inputs. The MPMC_DoneInit signal ensures that the MPMC memory controller is successfully initialized before attempting to read and write from memory. The frame_done signal is used to check if the video_switch module has successfully switched to the other camera and the frame data is ready to be read. This input signal was used to add a new waiting state in the FSM that blocks processing until the signal is high. The block still provides the X and Y coordinates as outputs. However, these are unused and the coordinates are now output through an FSL. 12

13 The color detection module requires that color bounds be provided from the Microblaze before processing the frame. The color bounds are not read through the FSL but through memory. This is done to reduce the amount of work that the software needs to do, as the game code can run a bit slowly. If the bounds were read through the FSL, the software would have to continuously write the bounds to the FSL, whereas once the bounds are written to memory, the software does not require to keep updating the FSL. Therefore, when the game code is first initialized, the code writes the various color bounds to memory. These bounds are: Lower Bound of Red for Color 1 Upper Bound of Red for Color 1 Lower Bound of Blue for Color 1 Upper Bound of Blue for Color 1 Lower Bound of Green for Color 1 Upper Bound of Green for Color 1 Together, these bounds set the range of color to be detected. These bounds are repeated again for the second color, but this is unused by the project. Once the color bounds are read, the module processes the entire frame stored in memory, finding the first and last points of a certain color in the entire frame. With these two points, the centre of the color is block is calculated. These coordinates are then passed to the FSL bus to be read by the Microblaze. 4.3 Fast Simplex Link (FSL) The FSL is a unidirectional bus that allows fast communication between both ends through a FIFO. The FIFO depth can vary greatly, allowing flexibility in the use of the FSL. It also supports synchronous and asynchronous modes, which allows the master and slave sides to have different clocks [1]. For the project, the FSLs are working synchronously, and they have a depth of 16, which is the minimum if the FSL is working synchronously. Two FSLs are used for the project, one for each color detection core. 4.4 Video to RAM The original IP comes from the Video to RAM project given. The hardware was modified to switch video input into two separate addresses in the DRAM. The Video to RAM is started by asserting frame_1_start or frame_2_start signal. Both frame_x_start and frame_x_done signals are connected to Video Switch IP core. Once a start signal is asserted the timer was synchronized to VBLANK signal and counts the number of lines written to memory. Once the entire frame is written, frame_x_start is deasserted and frame_x_done is asserted. After the operations are completed the block waits for another start signal to be asserted. 4.5 Video Switch This IP block was newly created for this project. It contains a PLB bus slave and 4 user addressable registers. This block is used to relay signals between the Microblaze processor and the Video to RAM block. The software in the processor is configured to write and read user addressable registers in the Video Switch IP core, which in turn are translated to signals frame_x_done and frame_x_start and 13

14 connected to the Video to RAM block. To start writing a frame the software writes 1 into bit 0 or bit 1 of C_BASEADDR + 0x4 for frame 1 or frames 2 respectively. Once the frame is finished 1 is read from bit 0 or bit 1 of C_BASEADDR. The actual video switching is done in software by reprogramming the Video daughter card through I2C. For this project we used green and yellow RCA connector on the daughter video board. 4.6 Plb_tft_cntrlr XPX TFT Controller v1.00a from Xilinx IP catalog. The XPS TFT controller is a hardware display controller IP capable of displaying 256k colors connected on the PLB. Used to interface with VGA monitor to display graphics.[2] 4.7 Xps_interrupt_controller XPS interrupt controller 1.00a from the Xilinx IP catalog. Concentrates multiple interrupt inputs from peripheral devices to a single interrupt output to the system processor. Registers for accessing interrupt data are accessed through a slave interface for the PLB. [3] 4.8 Plb_uart SPS UART lite v1.00a from the Xilinx IP catalog. Connects to the PLB and provides the controller interface for asynchronous serial data transfer, interfaces with PLBV46. Used for showing debug data over RS232 to a secondary PC on HyperTerminal. [4] 4.9 Xps_timer XPS timer/counter 1.00a from the Xilinx IP catalog. A 32bit timer module that attaches to a PLB bus. Produces an interrupt at a set time interval through interrupt controller. [5] 4.10 Debug Module Microblaze Debug Module v1.00d from Xilinx IP catalog. Enables JTAG based debugging on Microblaze processors. Used through the XMD interface to debug and step through code. [6] 4.11 LED debug mux LED debug MUX v2.01a. Debug multiplexor output for LED debugging Clock Generator Clock generator v1.00a from Xilinx IP catalog. Instantiates a DCM module, PLL module, BUFG insertion, automates DCM and PLL reset sequence determination and connection. [7] 4.13 MPMC MPMC v4.03a from Xilinx IP catalog. Fully parameterizable memory controller that supports SDRAM/DDR/DDR2. Provides access to memory for one to eight ports, selected from PIM that permits connectivity into a PowerPC and Microblaze processor using Core Connect PLBv4.6 and the NPI, and memory interface block PIM. Supports SDMA, video frame buffer controller, error correcting code and performance monitoring. [8] 14

15 5.0 Description of Design Tree The design tree structure of our project is detailed below. /BoxingGame/- main directory Group Report /BoxingGame/doc - Group Report Software /BoxingGame/sw/DrawText.h - hard coded letters and numbers for drawing text on VGA /BoxingGame/sw/gamedetect.c - main game code /BoxingGame/sw/List.h - some structures for game data /BoxingGame/sw/VGA.h - VGA functions /BoxingGame/sw/constants.h - constant definitions Hardware /BoxingGame/pcores/led_debug_mux_v1_00_a - debug LED core /BoxingGame/pcores/paddle_detector_fsl_v1_00_a - the video processing module for color detection /BoxingGame/pcores/video_switch_v1_00_a - video switch module, used to switch between cameras /BoxingGame/pcores/video_to_ram_v1_00_a - modified Video to RAM module to handle 2 cameras Block Diagram /BoxingGame/blkdiagram - EDK generated block diagram All other files/folders are standard Xilinx generated files/folders. 6.0 References [1] LogiCORE IP Fast Simplex Link (FSL) V20 Bus (v2.11c), Internet: April 19, 2010 [April 6, 2012] [2] "Xilinx Thin Film Transistor (TFT) Controller (v1.00a)", Xilinx EDK v SP3, July 21, [3] "XPS Interrupt Controller (v1.00a)", Xilinx EDK v SP3, July 22,

16 [4]"XPS UART Lite (v1.00a)", Xilinx EDK v SP3, July 18, [5]"XPS Timer/Counter (v1.00a)", Xilinx EDK v SP3, April 21, [6]"Microblaze Debug Module (MDM)(v1.00d)", Xilinx EDK v SP3, June 25, [7]"Clock Generator (v2.01a)", Xilinx EDK v SP3, July 28, [8]"Multi-Port Memory Controller (MPMC)(v4.03a)", Xilinx EDK v SP3, July 30,