MotionPro Team 2 Delphine Mweze, Elizabeth Cole, Jinbang Fu, May Oo Advisor: Professor Bardin Midway Design Review 1
Project Review A projected game that can be played on any flat surface A step towards more interactive gaming Our project: the machine detects motion of a finger to control the game http://www.fitness-gaming.com/images/markets/home_fitness/1107_lumo_play_interactive_projector_no w_tracks_childrens_toys/022_lumo_play_interactive_floor_children_room.jpg 2
Address PDR Concerns Met with Professor Holcomb in October Clarified our system design and revised deliverables Switched process from edge detection to background subtraction Project significance Demonstrating real-time system Societal impact Dance game, YMCA, gym class Hologram video gaming Educational archaeology game at museum Alternatives IR filter Touch screen 3
High Level Block Diagram Raspberry Pi Camera (Background Subtraction) (Hand Motion) Raspbian OS Speaker Image Processing Coordinate Detection Class Interfaces + Ultrasonic Sensors (Hand Location) Main() Game Program Projector Power Supply 4
Reasons For Ultrasonic And Camera Ultrasonic Sensors Camera/Image Processing Pro: Can give an exact distance Pro: Fast enough to detect (in cm) user while playing game Pro: Detects obstacles; not affected by game projection Pro: More suited for motion detection Con: Too slow to detect user while playing the game Con: Not good at detecting still hand Con: Susceptible to noise Con: Can t give exact distance 5
Promised MDR Deliverables May Functioning background subtraction Elizabeth Detection of hand position Delphine Fu Programmed snake game Interface simulating actual game inputs and outputs 6
Background Subtraction: May MacBook Pro Camera MDR Deliverable: Functioning background subtraction using laptop Purpose: User s finger motion needs to be detected by pi camera in order to determine the direction of the fingers as a next step Status: Completely functioning Pi Camera 7
Background Subtraction: May Study Solution Average human reaction time: 0.25 seconds to a visual stimulus* Theoretically, estimated fps can be as low as 4 fps Fastest reaction time: 0.101 seconds Use OpenCV and C++ to run background subtraction Use MacBook Pro camera to test the result Goal At least 10 fps ~ 0.1 seconds each frame CMake Use CMake to build a cross platform OpenCV Add opencv libraries CodeBlocks (IDE) Xcode (IDE) on Raspberry Pi *https://backyardbrains.com/experiments/reactiontime#prettyphoto 8 http://wordpress.as.edu.au/jhunt2/files/2014/02/i.chzbgr-245x300.jpg
Experiment Using Different Resolution: May 9
Background Subtraction: May Background Frame Video Detect Motion Frames Threshold Display Foreground Frame Choosing Background Subtraction parameter: Threshold variable: is chosen based on the standard deviation of gaussian distribution High threshold variable 10
Hand Coordinate Detection: Elizabeth MDR Deliverable: Detection of position of a hand using ultrasonic sensors Purpose: User needs to press menu buttons to start the game and to change modes Status: Completely functioning Demonstration at end https://www.aimagin.com/media/catalog/product/cache/1/image/9df78eab33525d08d6e5fb8d27136e95/h/c/hc-sr04-02.jpg 11
Visual Diagram http://csharpcorner.mindcrackerinc.netdna-cdn.com/uploadfile/167ad2/how-to-use-ultrasonic-sensor-hc-sr04-in-arduino/images/hc-sr04.jpg 12
Timing Diagram 10 μs 40 khz 200μS <5.682 ms Total: <5.703 ms https://playground2014.files.wordpress.com/2014/01/hc-sr04_datasheet.jpg 13
Multiple Receivers Set a common Trigger signal Multiple Echo pins Sensor with smallest recorded distance indicates the sensor the obstacle is closest to 14
Snake Game: Fu Function of the Game 15
Snake Game: Fu Process of the Game 16
Snake Game: Fu Make it better Add more pictures Fruit Snake Improve the algorithm Decide which wall the snake may hit before the death of the snake Adjust the resolution of the game to make it portable 17
Game Interface: Delphine Goal: To build a well functioning interface to facilitate subsystem communication. Desired Specs: Negligble delay < 1 second Fast Processing Effective multithreading and synchronization Game inputs written in C++ and python. Current cross communication done using xml-rpc 18
Block Diagram Apache XML-RPC Library - HTTP used as Transport Layer https://technet.microsoft.com/en-us/lib rary/cc738291(v=ws.10).aspx 19
Block Diagram Snake Game Prototype (Java Client) Coordinates (Python Server) Apache XML-RPC Library EventSimulator Thread (robot) Main Thread (run snake) Camera Data (C++ Server) 20
Alternative Class Implementation Google Protocol Buffers: Enables user defined message types between languages Sophisticated algorithm in less code to reduce processing latency. Use of a communication model that allows the server to automatically send inputs when done with processing 21
CDR Deliverables May Background subtraction on Raspberry Pi at least 15 fps Translate the detected motion into right, left, up, down directions Fu Power supply design Circuit design Elizabeth Integrate ultrasonic data with rest of system Printed Circuit Board (PCB) layout Delphine Containers for synchronized data Integrated System 22
2016-2017 Schedule 12/14 1/18 2/1 2/15 3/1 3/15 3/29 4/12 4/26 May: Background subtraction on Raspberry Pi Elizabeth: Integrate ultrasonic data with Delphine s interface Fu: Power Supply Design Delphine: Determine direction from May s data May: Housing Elizabeth: PCB Layout Fu: Circuit Design Delphine: Containers for synchronized data All: Complete integrated system 23
Demos Deliverable #1 (May) Background subtraction using laptop Deliverable #2 (Elizabeth) Detecting hand position using multiple ultrasonic sensors Deliverable #3 (Fu) Showing the Snake game programmed from scratch Deliverable #4 (Delphine) Demonstrating game inputs/outputs 24
GameOver 25