ECE532 A/V Touch Miaad S. Aliroteh, Joe Garvey, Ben Hare 4/9/2012

Transcription

1 ECE532 A/V Touch Miaad S. Aliroteh, Joe Garvey, Ben Hare 4/9/2012

2 Table of Contents 1 OVERVIEW Motivation Goals Block Diagram Brief Description of IP OUTCOME Results Evaluation of Acceptance Criteria What would we do differently if we could start over? Improvements/Future Work PROJECT SCHEDULE Milestone Chart Discussion of differences DESCRIPTION OF BLOCKS video_to_ram xps_tft Microblaze touch_audio finger_detector Software DESCRIPTION OF DESIGN TREE TIPS AND TRICKS REFERENCES Appendix A: Proposal Features and Acceptance Criteria Appendix B: Resource Usage System touch_audio... 31

3 LIST OF FIGURES Figure 1: System block diagram... 5 Figure 2: List of project IP cores... 6 Figure 3: Pixel Encoding for xps_tft [5] Figure 4: The four key signal processing stages of the touch_audio IP [1] [2] [3] Figure 5: Finite Impulse Response filter using serial-parallel mutiplier-adders Figure 6: Piplined FFT Architecture [2] Figure 7: touch_audio analysis block Figure 8: Driver/game interactions Figure 9: Game execution flow... 26

4 1 OVERVIEW 1.1 Motivation The main inspiration for our project was the current world wide use of smart phones, and the growing appreciation for being able to operate a computer by tapping and dragging a finger along the screen. We wanted to be able to turn any monitor into a touch screen, allowing for easier use of any applications on the computer and making it more intuitive to use the computer. 1.2 Goals As stated above we wanted to implement a system which would be able to turn any regular computer monitor into a touchscreen. Our initial goal was to be able to use a normal computer with the system, essentially having our project interfacing with it like a mouse would, but we realized early on that it would be impossible to implement a mouse interface in the time provided. Therefore the goal changed to being able to play a game, running off of the board, on a screen using our project. The goal for the system was to use a combination of a video camera and microphone to detect finger taps and drags, and then transmit the location of the tap or drag to the MicroBlaze processor. The MicroBlaze would then use this input to play a game which would be displayed on the monitor. The game in and of itself was not a goal, as the hopeful extension of the project is to interact with an actual computer, but it was necessary to design something to showcase that the underlying hardware was working. To this end a variation on the game Fruit Ninja was to be designed, whereby users would cut fruit on the screen and receive points.

5 1.3 Block Diagram Figure 1: System block diagram 1.4 Brief Description of IP The various ip cores in the system and their versions are shown in Figure 2 below. Further details are provided in the following table.

6 Figure 2: List of project IP cores IP Function Origin On Chip IP microblaze Soft processor on which all of the software runs. Xilinx lmb_v10 Local memory buses used by the microblaze to access its BRAM. One for instructions and one for data. Xilinx plb_v46 Bus used through the design to communicate between various IP cores. There are four PLBs in our design, one for each IP core that acts as a master. These are: the microblaze, the video_to_ram core, the xpx_tft core (video output), and the finger detector core. For the most part these buses are used to communicate between the listed IP core and the MPMC although there are exceptions in which cores communicate directly with each other without going through the memory. Xilinx lmb_bram_if_cntlr mpmc bram_block Memory controller for the processor's BRAM that communicates with the processor over the lmb_v10 busses. There are two of these, one for instructions and one for data. Multi-ported memory controller. Provides a parallel way for the various cores to access the off-chip RAM. BRAM for the microblaze processor. All of the microblaze's software and data is stored here. Xilinx Xilinx Xilinx

7 finger_detector Custom core based on the Virtual Pong project's "paddle_detector" core. Determines the location of a finger in a video frame read from memory and writes that location to memory. Custom mdm JTAG debug module for the microblaze. Xilinx proc_sys_reset Reset block which receives an external reset signal from a pushbutton and produces a variety of reset signals for use within the system Xilinx touch_audio video_to_ram Receives video input from the video decoder chip and saves it to memory. xps_iic Used to communicate with the on-board video decoder chip. Xilinx xps_intc Interrupt controller. Used by the touch_audio to send Xilinx interrupt to the processor. xpx_tft Reads pixel values from memory, formats the data and sends to the VGA output. Custom Virtual Pong Xilinx xps_timer Timer used by the processor to gather pseudo-random data. Xilinx xps_uartlite Used to send data back to the host PC over the serial-to-usb Xilinx cable. clock_generator Receives a signal from the on-board oscillator and produces a variety of clocks used by different cores in the system. Xilinx Off Chip, On Board IP ADV7183B Video decoder chip. Analog Devices KVR266X66C25/512 SDRAM DIMM Kingston LM4550 Audio codec that performs AC97 analog functions. National Semiconductor FMS3818KRC_NL Triple Video D/A Converter Fairchild semiconductor Off Board Peripherals PowerShot A2200 Camera Canon mic Piezo contact microphone CUI Inc 2 OUTCOME 2.1 Results We were able to create a working touchscreen system with a video game to show off the work. The finger detection system is able to find the location of a finger wearing a red finger glove through the camera and accurately transmit this information to the processor. The audio system is a bit more unreliable, being that it does still sometimes detect a swipe or tap when there is none, or fail to detect a swipe or tap, but these issues have been reduced as much as possible and the game is still playable. The swiping logic with respect to the finger detection logic is flawless and always detects the swipe, a considerable achievement. The video game accurately showcases the work with an enjoyable and

8 playable game, whereby in an adaptation of Fruit Ninja fruit fall from the top of the screen to the bottom before being cut by the player, and if the player misses five fruit then the game is over, but if they reach 2000 points through cutting fruit then it advances to the next level. The overall FMax for the entire system is MHz. More detailed results, comparing to our initial goals, are included in the following sections. 2.2 Evaluation of Acceptance Criteria Note: Our original goals included in the design proposal are included in Appendix A: Proposal Features and Acceptance Criteria, and can be used for reference throughout this section Video System Goals The original goal was to do real time processing, at a reasonable frame rate and resolution, to detect a user's finger with sufficient accuracy and precision. Our resolution goal was to be able to handle an input of at least 320x200, which was exceeded as we are able to support an input resolution of 640x480 pixels. In terms of frame rate we wanted to operate at at least 24 frames per second. We met this goal, as the IP core that we used to handle video input (video_to_ram) can operate at much higher frame rates [4]. The accuracy goal was more of a heuristic, being that the system should not perceive erroneous locations so often as to make the game unplayable. Since the game is very much playable this goal was met. Our precision goal was to not have the system ever miss by more than 2 cm on the screen, which we were able to accomplish Audio System Goals The original goal for the audio system was to be able to detect touches and drags on the screen, and be able to distinguish between them and background noise in real time. The goal for the rate was to be able to detect at least 12 events per second, and as the system is able to handle at least 50 events per second this goal was met. The accuracy goals we had were to be able to correctly distinguish between taps, drags, and background noise. This is possible, but because of the differences between monitors cannot be guaranteed, and it requires a good deal of calibration in order to make work. Because of these problems this goal was not completely finished, but the game is still playable under the right conditions Video Game Goals The goal for the video game was to create an interactive game which used touches and drags, along with the location, as important features of the game. This was completed, as the Fruit Ninja game uses taps for menu selections, and drags for cutting the fruit, and the actual location of the finger is important in both instances for determining exactly what was selected or cut. For the difficulty goal, which was to use both shapes and colour in the game design, this was completed as the fruit game uses different coloured and differently shaped fruit in its execution. 2.3 What would we do differently if we could start over? One of the most important changes we would make if we were to start over would be to set up our system to store all of our code and data in external RAM rather than in the Microblaze s BRAM. The limited size of the BRAM turned out to be a great challenge throughout the project as our code was not initially small enough to fit in it (even at the maximum allowable BRAM size for a Microblaze). As a

9 result various parts of our address space would overwrite each other leading to very strange, unpredictable, and often unrepeatable bugs. If we were to start this project over knowing what we know now, we would have added the ability to filter pixels based on differences in colour right from the beginning. We wasted many hours, over the course of three weeks, calibrating the video input every time we switched locations or even when lighting changed. Alternatively, we may have implemented the algorithm in YCbCr rather than RGB as it provides a more intuitive representation of colours that separates out the impact of lighting. Initially we had planned to average the FFT output to obtain a spectrum that contained less noise. If we were to do this project next year, when the Virtex 5 s are available, we would add this part as it would likely make much of the calibration problems go away without running into area constrain problems. In addition, we had initially planned to do more complicated analysis of the spectrum rather than finding the frequency containing half the spectrum energy. We wanted to create a spectrum signature for a tap and swipe signal respectively and then compare how well a given spectrum matches these signatures. Essentially, these signatures would act like a finger print for a tap and swipe signal containing more detailed information that could be used to more accuracy detect taps and swipes and better distinguishing them from noise. Of course this complex analysis would likely not fit into the Virtex-II so this would only be possible on a Virtex-5. This is something we had not anticipated at the beginning of the year. In the software, we would have begun coding it in C instead of C++, saving future time as it turned out to be very difficult to compile both of them together and we had to switch it over to C midway through the project, with obvious problems associated with it. We would also have recommended the use of multiple microblaze processors, if only so that we would have had a larger text region to use as we are currently running short of room and have not been able to implement everything desired as a result. 2.4 Improvements/Future Work There are a few potential improvements that could be made to the system: 1) Improved Finger Detection: Being able to detect a finger by itself, without needing to use a red dot on a glove. This would make using the overall system easier, as a user could simply touch the screen with a naked hand, and it would allow more than one user to keep switching back and forth with the system. The best way to implement this would likely be to use a machine learning algorithm which can recognize a finger and put it into hardware. The code for it it probably too expensive to do in real time in software, although this could investigated as well. Unfortunately, since our system already uses up most of the Vertex II FPGA, in order to implement the audio detector a more modern FPGA would likely be required. 2) Interfacing with a Computer: As stated earlier, the hopeful future extension of this work would be to be able to use it with an actual computer. In this application the system would function as a mouse, whereby the location and tap/drag would be converted into a location and click on a mouse interface. This would likely be a considerable amount of work, probably constituting an

10 entire new project but would result in a fairly impressive complete project. 3) Improving the Audio System: Currently the audio system is not perfect, and calibration is very particular to each monitor it is used with. This would be a problem in any kind of commercial model of the system which would hope to be easily applicable to any screen. ************ 3 PROJECT SCHEDULE 3.1 Milestone Chart The chart on the following page depicts our schedule and progress over the course of the project. In the first column is the original milestone that we had set for that week in the project proposal. Over the course of the project, in consultation with the TA, we frequently revised these milestones. Thus the second column indicates the revised milestone for that week. These objectives were the milestones that we tried to meet each week, and it was against these milestones that our weekly progress was judged. Finally, the last column describes what was actually accomplished that week. In many cases, we met the milestone as described in the Modified Milestone column, and so these cases we have simply marked Met. If there was divergence from the modified milestone then it is explained in this column. If a field is left blank it is because that person had no milestone for that week.

11 Week Date Original Milestone (from proposal) Modified Milestone Actual Accomplishment 1 08-Feb-12 All: Ben: Joe: Get the VGA output to display on the monitor using the video system provided on the website. No change Met Miaad: 2 15-Feb-12 All: Ben: Have the game planned out, both what the game is going to be and what the components will be No change Met written in. Joe: Using the laser pointer project from one of the previous years, detect the location of red pixels in the video input. Met Miaad: Model step by step decimate-by-4-and-fir and FFT operations in MATLAB. No change 3 22-Feb-12 No milestones due to reading week 4 29-Feb-12 All: Ben: Complete low-level c classes for the game. Implement the underlying code before graphics are added. Joe: Detect the location of pixels of a particular colour on the video input. Write the entire initial version of the game (some features can be added later). Create h files that define the interface functions between the game and the video processing (at both input and output). Met Partially met: Began writing main game using video driver API. Met. Also created dummy c files that provided pseudo random input to facilitate testing.

12 5 07-Mar-12 All: Ben: Miaad: Implement FIR and compare with MatLab: BRAM data in -> FIR Verilog Block -> BRAM result out. Joe: Miaad: 6 14-Mar-12 All: Begin integration of hardware and software components. Ben: Game will have the acceptance criteria completed so that the drivers and interrupt service routines can interact properly. Implement FIR filter and get AC97 audio module working. Make sufficient improvements to the game in order to make it playable. Port the game to the microblaze. Debug the AC97 and FIR interface to get the mic working. Port the game to the microblaze and get it running on the board. Met Partially met: the game was improved to the point that it was playable on a PC, but was not ported to the microblaze. Met. Partially met: the game compiled successfully for the microblaze but was not running correctly because its.bss section was too large for the BRAMs. Joe: API the game uses to access video input should be written, but since the underlying hardware is incomplete will not be completely working. Miaad: Interface used between the software and the audio block defined, but not completed. Implement FFT and compare with MatLab: BRAM data in -> FFT Verilog Block -> BRAM result out 7 21-Mar-12 All: Integration continued on the hardware side. Detect a coloured dot (from the fingertip of a glove) on the screen and print the location of that dot in the input video frame to the terminal using a microblaze. Finish FFT and demonstrate it. Define driver API for audio detection core. Met Partially met: the FFT was finished but there was no way to demonstrate its functionality until the next week's milestone was completed. The driver API interface was defined.

13 Ben: Graphics and user interface for the game completed. Joe: Video decoder is able to detect a clump of pixels of a particular colour (i.e. a gloved finger tip), and communicate the location using the driver's API. Miaad: 8 28-Mar-12 All: Ben: Video game completed; small additional features may be added but they should not be hardware dependant. Joe: Video processing can work in real-time, anything that needed to be migrated from software to hardware has been. Miaad: Tune calibration and pattern recognition for the audio block in MatLab then start to implement the control logic in Verilog. Combine game with video input/output and finger detection. Use a pushbutton to simulate the audio detection core. Combine game with video input/output and finger detection. Use a pushbutton to simulate the audio detection core. Add a variety of new features to the game: a title screen, better fruit shapes, and the user's score. Improve the fruit swipping accuracy to 100%. Create executable Matlab model of the core. Finish analysis block in order to differentiate between swipes and taps. Demonstrate functioning of FFT. Met Met Met Partially met: the accuracy was greatly improved but was still less than 100%. Met 9 04-Apr-12 All: Begin final system-wide testing and debug, no new features should be added after this point.

14 Ben: Add the following game features: difficulty levels, advancing to a new level at a certain score, bombs. Partially met: different difficulty levels and the automatic advancement to the next difficulty level at 2000 points was added, bombs were not. Joe: Integrate audio detection core with the rest of the system. Fix video output colour. Add colour difference filtering to finger detection core. Partially met: fixed the video output colour and added colour difference filtering to the finger detection core. The audio detection core was integrated with the rest of the system but was only partially functional. Miaad: Complete audio block control logic block and test it. Finalize PLB interface to Custom Audio IP peripheral. Integrate audio detection core with the rest of the system Apr-12 All: Present the final demo. Finish integration Ben: Fix outstanding game bugs and improve consistency. Joe: Miaad: Improve fruit swiping accuracy. Finish integrating audio core with the rest of the system. Improve tap/swipe differentiation accuracy. Partially met: The audio detection core was integrated with the rest of the system but was only partially functional.

15 Some of the most significant changes to the schedule and the challenges that resulted in them are worthy of further discussion. These are presented below. 3.2 Discussion of differences Ben Working in C vs C++ I had originally coded all of the game in C++, as I have greater experience with it, and I enjoy working with it more as it has better features. This worked initially, as I was only working from my command line, compiling through a Makefile and not through XPS, but as soon as I moved the code into XPS and used their compiles I ran into problems trying to compile the C drivers from the Video to Ram system together with the C++ code I had written. The compiler used in XPS for Microblaze, mb-gcc, advertises being able to compile C with GCC and C++ with G++ but I was unable to get it to do this with the different kinds of files together. Because of this I was forced to switch over to C midway through the project and work with that Memory Issues In my initial version of the game I had created a global variable to store the state of the screen, essentially storing three integers per pixel to store those colour values. When I switched over to working off of the board I ran into problems as this construct was far larger than the space I had available for global variables. I was forced to switch over to only storing the locations of the fruits, and then just eliminating them and redrawing them each cycle. As well we ve had issues with fitting all of the drivers and code into the text region of the processor, and if I could do it over I would have added another processor to allow for more code to be written Joe Calibration Calibration of the finger detection core was a far bigger challenge than it was thought to be initially. As a result, each time we moved settings we had to spend a few hours recalibrating the colour detection for the new lighting conditions even though this recalibration could be performed in software. It was eventually decided, after discussion with other groups, that the solution to this issue was to add functionality to the finger detection core to be able to filter out pixels based on differences between RGB values rather than simply based on absolute ranges. This is reflected in the milestone added for April 4 th regarding this feature Accuracy My initial schedule overestimated how long it would take to get the finger detection core working and did not account for time spent improving its accuracy. In the initial schedule I was not to have the finger detection core complete until March 28 th. In reality I had this core finished and integrated with the game by March 21 st but had to continue working on accuracy improvements for the next three weeks. Accuracy ended up being a much greater challenge than expected and as a result had a significant effect on the schedule.

16 3.2.3 Miaad MatLab Model I underestimated the amount of time needed to design a proper FIR filter that would make my system more resistant to audio noise and at the same time make it enhance signal differences when comparing taps and swipes. In order to help me design the FIR, I decided to put in a lot of time and create an executable model of entire audio system. This way I could evaluate the effects of different FIR architectures and their performance in the entire system as a whole rather than blindly designing an FIR low-pass or high-pass filter. Unfortunately, this took a lot of time for which I did not plan for. Consequently, I had to spend many sleep-less nights making up for this in addition to having future milestones modified. Fortunately, this executable model also gave me some intuition on the challenges involved with calibration Design Not Fitting I misunderstood the fact that although certain logic blocks can be efficiently implement in VLSI the same is not true for FPGAs. I had assumed that using a serial-parallel multiplier would take up a lot less area than using parallel multipliers. Although this is true in general, the built-in Mult18x18 block in the FPGA will always be more efficient in area and speed as they were designed to be. Because I came to this realization after having built a lot of my custom IP, I was hesitant to make changes. Nonetheless I had little choice, so I decided to change the FFT block to use Mult18x18 block as the FFT used the most multipliers and it was somewhat easier to alter. 4 DESCRIPTION OF BLOCKS Simple commercial blocks that were used off the shelf will not be detailed here. For details on those blocks, such as versions used, see 1.4 Brief Description of IP. 4.1 video_to_ram This block was originally created by Jeffrey Goeders for the Virtual Pong project [4] and a modified version of it was part of the video demonstration system provided by the TAs on Piazza. This core takes video data from the off-chip video decoder and writes it to the DDR via the MPMC. 4.2 xps_tft This core reads from the RAM via the MPMC starting at a given address and interprets the values it reads as a 2D array of pixels. It encodes this data as a video and sends it to the off-chip video D/A converter. This is a standard Xilinx core, but we have listed it here because an understanding of its interface was necessary for the video output drivers to work correctly. Specifically, it expects the data in memory to be arranged in a particular way as summarized in Figure 3 from the xps_tft datasheet [5].

17 Figure 3: Pixel Encoding for xps_tft [5] 4.3 Microblaze The Microblaze is a soft-core processor made by Xilinx. Although it is an off-the-shelf piece of IP it is fairly configurable, so we will mention our hardware configuration here. The software running on the Microblaze will be discussed later in its own section (4.6). Our microblaze was instantiated with the maximum sized BRAM (64K) and it uses this BRAM as both instruction and data memory. We gave it extended floating point support as we perform a number of floating point operations when detecting intersections of lines and when converting between the plane of the video input to the plane of the monitor. Finally, it also has an integer divide unit because we frequently perform divides in our code for the same reasons. The software 4.4 touch_audio The Touch_Audio custom IP builds upon existing AC97 audio IP to primarily implement detection of taps and swipes made on a monitor with an attached piezo contact-microphone. This functionality would be used in a game involving a touch interface. This IP uses the AC97 block to initialize the LM4550 codec to capture audio samples from the microphone as well as to output some simple tones, for optional game sound-fx, in addition to looping back microphone input. The IP can be described in terms of 4 key stages of signal processing (Figure 4): (1) Signal capture and serial conversion, (2) Signal condition and filtering, (3) Spectrum transformation, and (4) Spectrum energy analysis. This entire process as whole was also modelled in MatLab with Simulink serving as an executable model to help guide the design process. The first stage is accomplished mainly with the AC97 block plus some parallel-to-serial conversion needed for the next stage. Signal conditioning is achieved with a 32-tap FIR filter (Figure 5) which attenuates low frequencies, removes mid-frequencies, and amplifies higher frequencies. This structure was chosen based on spectrum characteristics of taps and swipes observed using a Piezo contact-microphone in MatLab. The actual coefficients were determined using MatLab s FDAtool. A simple FIR implementation consists of a chain of multipliers, adders, and delay elements (i.e. memory). In order to reduce area used on the FPGA a serial-parallel multiplier-adder architecture was chosen which would allow continuous real-time processing of incoming data. This filtering process makes the Touch_Audio block less sensitive

18 to noise (voices in the background) and enhances the inherent signal differences between taps and swipes. After the signal is properly conditions it goes through an FFT block to transform it into the frequency domain. This output is then fed through a magnitude-squaring block to obtain a spectrum of the signal. This permits the extraction of meaningful data from the audio signals to distinguish finger taps from swipes and other signals. This stage is based on the Cooley-Tukey FFT algorithm (aka FFT Butterfly). It too was first implement using serial-parallel multipliers in the hope that FPGA area use would be minimized. However in practice the use of FPGA logic block for arithmetic is much more expensive than associated VLSI implementation. Hence, it became critical to use dedicated multipliers, Mult18x18, blocks instead to reduce area despite their parallel architecture. A second revision was made to the FFT block using the Mult18x18 blocks. Unfortunately, a direct translation to these multipliers meant that the touch_audio block could not integrate with other blocks which also used Mult18x18 blocks as touch_audio would require too many of these blocks. Hence, a third revision was necessary were the FFT was implemented in stages that were pipelined to compute the final output (Figure 6). This utilized much fewer Mult18x18 blocks and could easily integrate with other IP. The final stage of the touch_audio IP is the block responsible for analyzing the computed spectrum and determining if a tap, swipe, or unknown activity is occurring. From prior modelling, it is known that taps generally have lower frequency components than swipes, and that background (voice) noise is in concentrated in between, specifically in the 3kHz to 7khZ range. Hence determining the median frequency which contains half of the energy in the signal serves as a way of categorizing it as a tap, voice/noise, or a swipe. The analysis stage takes magnitude-squared spectrum values from the previous stage and computes partial energies by accumulating these values from 0Hz to each frequency-bin in the spectrum. The accumulated value for 0Hz to the last frequency-bin is the total energy in the system, E31 in Figure 7. This value is compared with thresholds values, through subtractors and sign comparators, to determine if an audio activity has occurred or if the audio system is overloaded with too much activity. If there is activity, but no overload, then the median frequency-bin, fm, for which the energy from 0Hz to f m is about half of the total energy is computed. This is achieved using subtractors. Since all subtractors operate on serial inputs and generate serial outputs, it is sufficient to subtract from the accumulated energy for each frequency bin, Ei, the total energy, E31, delayed by one clock cycle which simulates a divide by 2 on the E31 signal path. Once this median frequency, fm, is determined it is compared to other threshold values which correspond to expected median frequencies for taps and swipes to make the final decision on whether the activity is a tap, swipe or unknown..

19 Figure 4: The four key signal processing stages of the touch_audio IP [1] [2] [3]

20 Figure 5: Finite Impulse Response filter using serial-parallel mutiplier-adders

21 Figure 6: Piplined FFT Architecture [2]

22 Figure 7: touch_audio analysis block

23 4.5 finger_detector The finger detector block is a custom IP block that is used to determine the location of a coloured fingertip in the video frame. This core was based on the Virtual Pong project s paddle_detector. Essentially, the interface that the paddle_detector used to interact with the PLB was left mostly unchanged but the internals of the core were rewritten. This core functions by first reading a number of filter parameters from memory, these will be described later. It then reads a frame from memory and tests each pixel read against the criteria gathered earlier. The location (x,y coordinates) of the first and last pixels that meet the specified criteria are saved. Finally, once an entire frame has been read it writes the two saved locations to memory. The processor can then read these locations from memory when it wants to know the location of the finger (see the software section for more information on how this is done). There are two classes of criteria evaluated for each pixel. First, a minimum and maximum intensity is specified for each of red, green, and blue. These criteria are used to set weak bounds on which colours are acceptable. This approach is not sufficient in itself to detect an object of a particular colour because it is incredibly sensitive to lighting conditions. To deal with this issue, the core is also passed parameters indicating a minimum and maximum difference between each pair of colours (red green, red blue, and green blue). In order for a pixel s location to be recorded it must satisfy all six of the colour bounds constraints and all six colour difference constraints. This approach is very effective at selecting objects of a particular colour in a variety of lighting conditions. For example, we noticed that a particular red finger tip glove that we used for testing could have a variety of red, green, and blue pixel values depending on the lighting, but its red value was always much higher than its green and blue values, while those green and blue values always tended to be very close together with the blue slightly higher.

24 4.6 Software Drivers Figure 8: Driver/game interactions In order to use the different hardware peripherals, various drivers were written for video input, video output, audio input, and the timer Video The video input driver communicates with the finger detector block by reading from the off-chip RAM at the location that is written by the finger detector. Since the finger detector writes the x and y coordinates of the first and last pixels that match the colour criteria, the software takes the average of those two coordinates as the actual location of the finger. This location is not sufficient though for use with the game, as it is in the coordinate system of the camera not of the screen. The driver thus also converts the coordinates from the plane of the camera to the plane of the screen using previously stored coordinates representing the corners of the screen. These coordinates are gathered at the beginning of execution through a calibration screen. During the same initialization routine that runs the calibration, the various threshold criteria used by the finger_detector core are written to RAM (and subsequently read by the finger_detector). Although the finger detector supports changing these values with each frame, they are currently only set the once by the driver as the game had no need for this functionality. The video output driver takes in a location and set of RGB values for a pixel and then writes those RGB values, reformatted to the endianess expected by the video_output core, to RAM at the address that the video_output core will map to that location on the screen.

25 Audio The audio input driver reads a memory mapped IO address over the PLB, which then tells the game whether a tap or swipe was recorded Timer The timer is used to produce a pseudo random number for the parts of the game which require a random number Game The software implements a version of the popular Fruit Ninja game on the FPGA. The idea behind this game is that fruit will randomly appear at one edge of the screen and move to the opposite edge, and the player needs to swipe their finger through the fruit to cut it, causing it to be removed from the screen and incrementing the player's score. If the player manages to cut enough fruit then the game moves to a more difficult level, causing the fruit to move faster than before. If the player misses too many fruit at any level than the game is over. This game is all run on a single MicroBlaze processor. The game is implemented mostly in fruit_game.c, with all of the major drawing functions (for the fruit and letters) in fruit_draw.c. All of the driver calibration is done in main.c, after which the game is launched. The main game loop executes only once per complete game, and is only restarted by selecting start on the start menu. The inner loop has two segments: one for creating and moving the fruit, and one which tries to detect swipes of the fruit. The fruit creation/moving portion only executes every executions of the loop, and if a player misses five fruit then the game is over. It first checks to see if any of the fruit will be moved off screen in this iteration and if they will be, the fruit are removed and the remaining misses are decremented. It then deletes and redraws each fruit lower on the screen and randomly creates a new fruit at the top of the screen. The swipe detection portion checks to see if a swipe has started, and if it has it records the position. It also checks to see when that swipe has ended, and then using the starting and ending locations it checks to see if any fruit lie in the path of the swipe. Any fruit in the path of the swipe are removed and the player's score is increased. When a player reaches a score of 2000 or more the score resets and the game restarts on a higher level, essentially meaning that the creation/moving loop will execute more often, and the fruit move faster down the screen with each level. As well, the score, level, and number of lives remaining are displayed on the side of the screen.

26 Figure 9: Game execution flow 5 DESCRIPTION OF DESIGN TREE system.xmp XPS project File system.mss System Software Specification File system.mhs System Hardware Specification File data Includes the user constraints file system.ucf lib Necessary library files pcores Custom Hardware Peripherals o finger_detector Includes the code for the finger location detection module o audio_touch Includes the code for the swipe/tap detection module o video_to_ram Includes the video to RAM system needed for display sw Contains the software needed for drivers and the game o game Contains the files specific to the Fruit Ninja game fruit_game.h Contains the code for the main game loop and peripheries fruit_game.c fruit_draw.c - Contains the code for drawing out fruit, letters and numbers

27 o o o o o o o o o o o main.c Contains the calibration function calls and then the main run call audio_input.h Contains the highest level of the audio system drivers audio_input.c AudioControl.h Contains the parts of the audio drivers which interact with hardware AudioControl.c timer_driver.h Contains functions for interacting with the timer periphery timer_driver.c video_input_driver.h Contains driver functions for finding out the location of the finger video_input_driver.c video_output_driver.h Contains driver functions for displaying pixels to screen video_output_driver.c makefile Not used in the overall system, only used for compiling the code outside of XPS 6 TIPS AND TRICKS Store your software/data in off-chip RAM rather than in the Microblaze s BRAM so that your program size is not limited by the (very small) size of the BRAM Reduce area in ISE projects: o Right click on the synthesis command for the top-level module option the optimization options menu o Select advanced options o Use the Optimize primitives option o Do not keep hierarchy o Increase fan-out setting from default 500 to 4096 If generating clocks less than 32MHz o You cannot use Clock_Gen or DCM as they don t go down so low o Create the clock using counters to divide a faster clock then buffer it with the Verilog syntax: BUFG(.i(slow_clk),.o(slow_clk_buffered)) When simulating in ModelSim you can copy signals from the wave window and paste them into the text-editor section of ModelSim o This can be used to setup.do files quickly rather than trying to figure out how to go down levels of hierarchy to access signals For EDK projects, the.scr files in the synthesis folder contains synthesis optimization options o These files are regenerated by EDK so changing them does NOT change the optimization setting o There is no clear way of setting optimization settings in EDK 10.1 but later versions have a way to do this as claimed by Xilinx When generating PCores that use the PLB bus connected to microblaze processors o If your core is not setup correctly to provide acknowledge signals to the bus then when a microblaze debug module tries to connect to microblazes over PLB and reset it, it cannot do so properly.

28 o o o When launching XMD it will give an error that it cannot connect to microblaze This error will be mysterious as at first it seems there is a cable problem Only fixing the PCore will solve the problem Try looping RdEn and WrEn signals back to their associated acknowledgement signals to see if it fixes the problem Don t try to do the project through a Virtual Machine as there will be enumerable issues connecting to the board through it, and sometimes it won t recognize the cables are attached at all. This could be VM dependent, and if so then don t use the Oracle VM VirtualBox software. You re better off dual booting Linux or finding a version of Windows that works. Don t code in C++, use C instead as it is a hassle trying to get any C++ code to compile together with C code in XPS. Don t store the entire screen anywhere in your program. It won t fit. If you find you need to know the current value depicted on screen then read it out of the RAM instead.

29 7 REFERENCES [1] [2] Butterfly%203.jpg [4] Chirag Ravishankar, Durwyn D Silva, and Jeffrey Goeders. ECE 532 Pong Group Report. April 5, [5] Xilinx. XPS Thin Film Transistor (TFT) Controller (v2.00a) Product Specification. September 16, 2009.

30 8 Appendix A: Proposal Features and Acceptance Criteria Features Functional Requirements Acceptance Criteria 1 Does real time video processing to detect the location of the user's finger. 2 Detects the sound of the user's finger tapping or dragging on the video screen. 3 Displays a game on the screen that the user plays using the aforementioned method of interaction. Can work in real time at a reasonable resolution and frame rate. Is able to detect location of the user's finger with sufficient accuracy and precision. Can distinguish touches and drags on the screen from background noise and each other in real time. Uses the video and audio inputs to create an interactive game. The difference between touches and drags is important to the game, as is their location. a) Resolution: Can handle an input resolution of at least 320x200 b) Frame rate: Operates at least 24 frames per second c) Accuracy: The user does not perceive erroneous locations so often that the game is unplayable d) Precision: The distance between the center of the user's finger and where we believe the finger to be should be no more than two centimetres on the monitor we use for the demonstration a) Rate: Can detect at least 12 events per second b) Accuracy: The user does not perceive erroneous clicks or drags so often that the game is unplayable c) Accuracy: Clicks are not misinterpreted as drags or vice versa so often that the game is unplayable a) Difficulty: The game utilizes both shapes and colour in its display b) Utilization: Touches and drags both perform important, different operations within the game c) Utilization: The location of touches and drags is made use of in at least one facet of the game

31 9 Appendix B: Resource Usage 9.1 System Resource Used Total Percent BUFG muxs % DCMs % IOBs % 18x % RAMB16s % Slices % FFs % LUTs % 9.2 touch_audio The touch_audio was our largest single core. Its specs are given below ISE Resource Used Total Percent Slices % FFs % LUTs % 18x18 multipliers % XPS Resource Used Total Percent Slices % FFs % LUTs % BIOs % GCLKs % 18x18 multipliers %