Escort Bot for the visually challenged BTP report submitted in partial fulfillment of the requirements for the Degree of B.Tech. in Computer Science & Engineering on 18th April,, 2017 BTP Track : Research BTP Advisor : Dr. Pushpendra Singh BTP Students : Kartik Maji 2013051 Shubham Sinha 2013159 Taruvar Aggarwal 2013166
Student s Declaration I hereby declare that the work presented in the report entitled Escort Bot for the Visually Challenged submitted by us for the partial fulfillment of the requirements for the degree of Bachelor of Technology in Computer Science & Engineering at Indraprastha Institute of Information Technology, Delhi, is an authentic record of my work carried out under guidance of Dr. Pushpendra Singh. Due acknowledgements have been given in the report to all material used. This work has not been submitted anywhere else for the reward of any other degree. Place & Date: New Delhi, 18/04/2017 Kartik Maji Shubham Sinha Taruvar Aggarwal Certificate Certificate This is to certify that the above statement made by the candidate is correct to the best of my knowledge.... Place & Date:... (Dr. Pushpendra Singh)
Abstract The present invention, generally relates to a navigation aid for visually challenged people, and is essentially an escort bot system and method for providing navigational aid to the visually challenged and senior citizens. Keywords Blind, Escort, Robot, Waypoint, Technology, Data, Engineering, GPS, Research, Bluetooth, Communication, Ultrasonic, Arduino
Acknowledgement We would like to thank our guide and mentor, Dr Pushpendra Singh, for his constant support and guidance, without which we would not have been able to attempt this project. Ours is a comparatively unconventional research area in nature. It s the air of innovation at IIIT-Delhi and the encouragement of the professors that pushes students like us to take on interesting projects. We would also like to thank the Circuits and VLSI Lab for providing us required components. Work Distribution 1. Taruvar tried various design implementations and tried to make it as minimalistic as possible. After many tries we were able to bring it down to a two wheel design which is very handy. 2. Shubham upgraded the type of sensors in terms of accuracy and implemented an algorithm that keeps a check of 4 sensors removing the problem of ghost objects being detected by sensors. 3. Shubham and Taruvar calibrated both the sensors and the motors to fit the new design. 4. Kartik implemented dynamic mapping of routes even if the robot goes on a wrong route. 5. Kartik has also implemented voice recognition and KML mapping which gives us more accurate waypoints even on a curved path. 6. All of us have tried to learn new tools and technologies to help us in our project and make our product more user friendly 7. All of us contributed equally in compiling this report.
Index Student s Declaration 2 Index 5 1. Introduction 6 1.1 Conventional ways 6 2. Motivation 7 2.1 Research Problem 7 2.2 Related Works 8 White Cane 8 SmartCane at IIT Delhi 8 GuideCane 9 3. Product Figures 10 3.1 Fig 1 10 3.2 Fig 2 11 3.3 Fig 3 3.4 Fig 4 11 3.5 Fig 5 12 4. Product Description 13 4.1 Fig 1 13 4.2 Fig 2 14 4.3 Fig 3 14 4.4 Fig 4 15 4.5 Fig 5 16 5. Issues and Challenges Encountered 17 5.1 Design 17 5.2 Communication Technology 17 5.3 Movement direction and turns 17 5.4 Data Transfer between Android Application and Microcontroller 18 5.5 Indoor movement 18 5.6 Choosing the correct apparatus 19 6. Future Scope 20 7. References 20
1. Introduction 1.1 Conventional ways One of the conventional ways of assisting visually challenged people is guide dogs, (also known as service animals or assistance animals), which are assistance dogs trained to lead blind and visually impaired people around obstacles. However, training dogs for navigational and obstacle assistance is not an implementable solution. Availability of dogs, specially trained dogs, hinders progress in providing a reliable navigational aid to the visually challenged.
2. Motivation One of the biggest challenges for visually challenged / senior citizens or partially blind individuals is to navigate independently and do their day to day tasks independently. Humankind is trying to automate everything in today s dynamic world. We use applications on smartphones to do everything from ordering cabs to communicating with people around the world. But, even in this technologically advanced era today, visually impaired people face a lot of problem while commuting from one place to another. The difficulties range from finding the correct routes to avoiding colliding into obstacles on the way. Out of choice, they end up having to rely on other people around them for guidance. The aim of our project is to make this set of the population self reliant when it comes to their movement. Having worked together on many college projects assigned by professor and having a good understanding of the technology that can be used, we wanted to apply our engineering and problem solving skills to make a product that can actually make commuting much easier for the visually challenged and also make them self reliant. 2.1 Research Problem The fundamental problem was to make an automated robot that can take the present location and destination of user as input and plot the most ideal way and take the user along that way maintaining a strategic distance from all moving and nonmoving objects or humans/animals. For example if the user has to go from the boys hostel to the academic building, the bot will read its current location as IIITD Boys hostel and plot a path to Academic building going along the road and avoid any person/vehicle/stationary obstacle on the way. Also it should avoid all ghost objects and be able to traverse even a curved path without any problems. Accuracy of movement in an uncontrolled environment was also a big challenge.
2.2 Related Works 1. White Cane White stick is considered one of the best and most used travel help for the visually impaired. It is used to distinguish snags on the ground, uneven surfaces, gaps, steps, and puddles. The white stick is inexpensive, lightweight and it can be collapsed and concealed in a pocket. Which makes it easier to be carried anywhere. However, users need to invest at least 100 hours a generous "concealed" expense to learn the proper use of this stick effectively. 2. SmartCane at IIT Delhi SmartCane device is an electronic travel help which fits on the top fold of the white stick.it serves as an upgrade to the white stick and defeats its
shortcomings by distinguishing knee-above and hanging hindrances. It can even warn about the side of a truck, hanging fabric strings, jutting coolers or aeration and cooling systems, tree limbs etc. These objects don't have a considerable impression on the ground and can damage head or abdominal parts of the body. For safe commuting it is critical that they are warned about well before in time. 3. GuideCane GuideCane is a computerized travel aid for the active guidance of blind pedestrians. It is a device intended to help blind or visually impaired pedestrians to easily go around obstacles. The GuideCane is developed at the University of Michigan's Mobile Robotics Lab. It contains a long handle and a "sensor head" unit that is connected at the distal end of the handle. The sensor head is mounted on a steerable yet unpowered two-wheeled guiding hub. During operation, the client pushes the lightweight GuideCane in front of him/her. Ultrasonic sensors mounted on the sensor head recognizes the obstacles and steer the gadget around it. The client feels the directing nature as an exceptionally perceptible physical feeling through the handle and can easily follow the GuideCane s way.
3. Product Figures 3.1 Fig 1
3.2 Fig 2 3.3 Fig 3 3.4 Fig 4
3.5 Fig 5
4. Product Description 4.1 Fig 1 FIG. 1 illustrates a schematic representation of an escort bot system (100) As shown in FIG. 1, the system 100 includes an escort bot 102 and a mobile device 104. The mobile device 104 is a smart phone, and has a guide application installed on its operating system (for example, android, ios). The mobile device 104 is held by a visually challenged person The mobile device 104 is connected to the escort bot 102 via a network 106 Bluetooth. The guiding application is configured to access Google Maps and instruct the escort bot 102 to move in a particular direction, as per the destination of the user entered into the application. The escort bot 102 includes an elongated member 108 can, the elongated member 108 has an adjustable height Further, the elongated member 108 includes a proximal end portion and a distal end portion. The proximal end portion includes a handle. A user of the mobile device 104 can hold the escort bot 102 via the handle of the elongated member 108. The escort bot 102 further includes four ultrasonic sensors. 110 on the elongated member 108. The sensors are uniformly spread over the elongated member 108. Ultrasonic sensors 110 are configured to detect any object/obstruction in the desired paths of the escort bot 102. The ultrasonic sensors 110 are configured to measure the distance to an object by sending out a sound wave at a specific frequency and listening for that sound wave to bounce back. By recording elapsed time between the sound wave being generated and the sound wave bouncing back, the ultrasonic sensors 110 may calculate the distance between the sensors and the object in the path of the escort bot 102. Further, the ultrasonic sensors 110 are configured to notify a microcontroller (shown in FIG. 2), in case of objects detected in the path of the escort bot 102. Further, the ultrasonic sensors 110 are provided at different heights of the elongated member. This will ensure maximum coverage of area in front of the escort bot 102, so that the movement of the escort bot is not hindered in any way, and that the user doesn't collide with any obstructions. Further, the escort bot 102 includes a chassis 112, which is coupled with the distal end portion of the elongated member 108 via a spherical ball 114. The spherical ball is a 360 degree free rotating joint between the elongated member 108 held by the user and the escort bot 102. The ball 114 is configured to ensure that there is almost negligible resistance offered by the users stick to the escort bot s movement.
The escort bot 102 further includes wheels 116 coupled to the chassis 112. The wheels 116 include thick traction of rubber to provide increased grip, and give the desired motion, accuracy, and almost negligible slippage in the escort bot's movement The chassis 112 is cylindrically shaped. Diameter of the wheels is larger than diameter of the cylindrically shaped chassis. The large size of the wheels can easily move over smaller obstacles (e.g., small pebbles and stones) and hence, facilitate smooth rotational movement of the wheels. 4.2 Fig 2 FIG. 2 illustrates schematic diagram of an internal structure of the chassis 112. As shown in FIG. 2, the chassis 112 includes a Bluetooth module 202, a plurality of stepper motors 204, a microcontroller 206, a motor driver 208, and a battery (not shown in the figure). The Bluetooth module 202 is configured to communicate with the user s mobile device 104 and receives instructions including GPS coordinates via a Bluetooth channel. The instructions include direction of movement of the wheels 116 and speed of movement of the wheels. The instructions/commands including all waypoints (the current GPS coordinates and the next GPS coordinate), and are provided by the guiding application of the mobile device 104. 4.3 Fig 3 Fig 3 illustrates a block diagram of modules of the microcontroller. The Bluetooth module 202 is configured to send the received instructions to the attached microcontroller 206 (for example, Arduino Uno board based) for further processing via a serial bus. According to an embodiment of the present invention, the microcontroller 206 includes modules 210, which includes a process module 212 and a control module 214, as shown in FIG. 3. The process module 212 is configured to process the received instructions (including GPS coordinates) from the Bluetooth module. Further, the process module 212 is configured to process signals received from the sensors 110. The control module 214 is configured to control a motor driver 208 based on the processed instructions. According to an embodiment of the present invention, the microcontroller 206 is configured to control the power going into the motor driver 208 from the battery, based on the processed instructions by the process module 212 and objects detected in the path of the escort bot by sensors 110. Further, the motor driver 208 is configured to control the motors 204, which control the direction and speed of the wheels 116.
4.4 Fig 4 The fig illustrates exemplary waypoints 400 provided to the escort bot 102 and the escort bot 102 escorting a user towards his/her destination, according to an embodiment of the present disclosure. In an embodiment, the guiding application installed in the mobile device of the user, sends requests to Google Maps application programming interface (API) with required details (i.e. GPS coordinates of the starting location and the destination), based upon a destination entered in the application by the user. Further, the guiding application is configured to monitor current GPS location of the user. The guiding application is configured to deal with user s current location and dynamically update user s location with updated path to desired location. In case, the user has taken any wrong turn, the guiding application is configured to update the path, and find a suitable path towards the destination from the current location. In operation, the user needs to hold a handle of the escort bot 102. The Bluetooth module 202 of the escort bot receives instructions from the guiding application of the mobile device 104 and provides the instruction including GPS coordinates of next waypoints in the path to the microcontroller 206. The microcontroller 206 processes the instructions and controls the motor driver 208. The motor driver 208 controls the motors 204, which controls the wheels 116 for direction and speed of the wheels. In case of any obstacles found in the path of the escort bot by the ultrasonic sensors 110, the user is escorted along a different path by the microcontroller 206 to his destination so as to avoid any collision of the user with the obstacles. The wheels 116 of the escort bot 102 are moved by the motor 204 towards the destination based upon microcontroller commands, the escort bot 102 automatically proceeds towards the destination. As the user is holding the handle of the escort bot 102, the user is also escorted towards his destination, without requiring the user to push the escort bot (as the escort bot is powered by a motor). In an embodiment, when the user has reached his destination, the microcontroller 206 stops the movement of the wheels, thereby indicating to the user that s/he has reached his destination. In another embodiment, an audible instruction may also be provided to the visually challenged user by the mobile device when the user has reached the destination. Further, according to an embodiment of the present invention, the guiding application in the mobile device 104 of the user is configured to use keyhole markup language (KML) files (i.e., a file format that displays geographic data in a earth browser such as Google Maps). Those skilled in the art will appreciate that KML files provides better waypoints instead of waypoints given by Google maps API, as the Google maps API does not take in account the curves and turns
which are required by the escort bot 102 while escorting the visually challenged user. In an embodiment, the KML files may provide exact angles to the escort bot, and thus facilitate the escort bot in traversing a curved path with more waypoints and better accuracy. Those skilled in the art will appreciate that by using KML files, the guiding application is able to compute movement of the escort bot between a current location and the next expected location (provided by the guiding application) in a better manner. Further, in an embodiment, the guiding application may use pre-determined tools (for example, mapstogpx) to convert directions provided by Google Maps into a GPX file (i.e., a GPS exchange format file) and get the waypoints of the path to the destination, which are provided to the escort bot 102 via the Bluetooth module. 4.5 Fig 5 FIG. 5 illustrates an exemplary flowchart 500 for a method of guiding a visually challenged user from his current location to a desired location. Initially, at step 502, instructions are received by the escort bot from a mobile device. The instructions comprises GPS coordinates or waypoints of a path from a current location of the user to a desired destination of the user. At step 504, at least one object is detected by the ultrasonic sensors in the path of the escort bot. The ultrasonic sensors may measure the distance to an object by sending out a sound wave at a specific frequency and listening for that sound wave to bounce back. Further, the ultrasonic sensors notifies the microcontroller of the escort bot, in case of objects detected in the path of the escort bot. At step 506, the received instruction by the escort bot are processed by the microcontroller. The received instructions include GPS coordinates of waypoints in KML files, which provide exact angles to the escort bot, and thus facilitate the escort bot in traversing a curved path with more waypoints and better accuracy. At step 508, directional movement and speed of the wheels is controlled by the microcontroller based upon the processed instructions and the at least one object detected by the ultrasonic sensors. The microcontroller is configured to control the power going into the motor driver. The motor driver controls the motors, which controls the direction and speed of the wheels of the escort bot. At step 510, it is determined by the microcontroller if the user has reached his destination. In case the user has reached destination, the method concludes. Otherwise, the method returns to step 506.
5. Issues and Challenges Encountered 5.1 Design One of the foremost challenges was to decide the design of the whole setup. Our primary aim was to provide a guidance system through a device that is already used by the visually challenged. Hence, we decided to incorporate our system by tweaking their walking cane. Hence, it had to be of optimum size and weight which would not hamper their usual experience. To guide them in the right direction, we added 4 wheels at the bottom attached through a 360 degree rotation wheel to the cane. After the user inputs his/her destination through the phone app, the wheels start moving in the direction of the destination guiding the user holding the cane. The 4 wheels made the cane heavy and difficult to carry when not being used. We wanted to minimise our design and make it more portable as well as attractive in terms of a commercial product. After much deliberation and trials we have been able to get to a much more compact as well as user friendly design in which the bot has just two wheels and is easily portable. 5.2 Communication Technology We have used an Arduino Uno to control the servo s attached at the base. Once the user enters his destination, the mobile has to communicate with the Arduino so that movement at appropriate speed and angle can be initiated. At every coordinate the mobile feeds the Arduino with the angle that the robot should move in and that is constantly being updated in reference to the present location taken by the GPS of the phone. Hence, the communication needs to be stable, fast and low power consuming. We connected Bluetooth chip HC-06 to the Arduino board which sends and receives data from the mobile phone. 5.3 Movement direction and turns Google Maps API only returns values at GPS points where user needs to take a turn which is intuitive in cases where a user is capable of navigating. However, when we consider visually challenged people it fails in several cases such as : When the road starts to curve, the API does not return any information.
When their are more than two roads to choose from one in the same direction, the API return the same result. For eg Initially, our bot followed the boundary wall to commute in curved roads and followed it till the next turn or the desired location was reached. We have implemented waypoint navigation using the KML file from Google Maps and now we are able to input the exact angle of movement for the upcoming 5-10 meters of movement. 5.4 Data Transfer between Android Application and Microcontroller We are using bluetooth to communicate between the android application and the microcontroller. We faced several issues when we were setting up this connection. While the android app and arduino were both able to send data to each other sequentially, the received data on both of them was garbage. It was later that we discovered that the baud rate of both the appliances was different because of which the received data was null or did not make any sense. 5.5 Indoor movement The bot s movement inside the buildings is still very unreliable due to lack of GPS accuracy. We plan to implement SLAM algorithms so that our bot can perform basic navigational tasks in majority of the cases even indoors.
5.6 Choosing the correct apparatus Wheels We had to search for the appropriate wheels which had a large enough diameter, were made of thick high-traction rubber, had the correct shaft diameter and were not too costly. This would reduce wheel slippage and ensure that the motion of the wheels is not disrupted by road dust or pebbles. Motors Choosing the right motor was crucial to our project as it is important that we calibrate the direction of motion very precisely. The bot needs to move at the exact angle to follow the road accurately navigating around the obstacles at the same time and so, we required feedback from the motor in terms of the angle that it had turned from it original position. Hence, we have used stepper motors in the prototype. Using PWM we are able to program and ensure accurate angular movement. Proximity Sensors Obstacle detection is a major part of our project. Hence, the used apparatus has to be able to : Detect distance to the real obstacles accurately to avoid collision Not give false readings and/or create ghost objects so as to avoid unnecessary rerouting. Using sensors used in car reverse parking, we are able to judge this distance properly. Power Power is a major concern since we need to power the Arduino which in turn powers the motors, bluetooth chip and the sensors. Since, arduino s can provide up to 5V while normal motors typically require between 9-12 V, we need additional small size battery to power the motors as well. Hence, we have installed a 12V output LIPO battery in the cane
6. Future Scope We can sync this bot with Traffic Lights which can assist user to stop in case their is red light for pedestrians and cross the roads only when the traffic comes to a halt. This project can be further extended to create a network of IOT which can be used to collect continuous data from users and analysing the data to map less busy paths or with low probability of coming across obstacles. Since we are already using GPS, user can also get valuable feedbacks about their surrounding like nearby stores, restaurants etc. Hence making a system for better navigation of visually impaired people. 7. References The GuideCane A Computerized Travel Aid for the Active Guidance of Blind Pedestrians SMARTCANE - http://assistech.iitd.ernet.in/smartcane.php ArduinoBoardUno - https://www.arduino.cc/en/main/arduinoboarduno Android Developers - http://developer.android.com/guide/index.html HC-06 Datasheet - https://www.olimex.com/products/components/rf/bluetooth-serial-hc-06 /resources/hc06.pdf HC-SR04 Datasheet - http://www.electroschematics.com/8902/hc-sr04-datasheet/