Smart Assistive Armband Through Haptic Feedback For Visually Impaired.docx

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 1

INTRODUCTION People with visual disability need a constant assistance in their lives to carry out the daily routine. The assistance required may range from relying on other people for help to using various Electronic Travel Aid (ETA) assistive devices whenever needed. In order to make the blind independent and provide a smart assistance we aim to develop a smart wearable device which would guide the user in daily routine and also provide assistance to navigate the indoor as well as outdoor environment. Our project is a wearable device that greatly increases the independence of the blind. It is an armband and smartphone app combination, designed to help the visually impaired by integrating machine learning with haptic feedback. After waking the device from sleep, with a wake word or button press, users can summon the smart assistant to help them to identify products. It can detect the objects and then guide the users arm towards the object through haptic feedback. Through facial recognition, the device can even remember and identify faces to help the user recognize people he or she has met before. The device can be recharged on any standard Qi wireless charging pad. The device is a wearable solution that minimizes the daily problems faced by the blinds.

Dept. of ECE, KLECET, Chikodi

Page 1

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 2

LITERATURE SURVEY The necessity for developing a cost efficient assistive system for the visually impaired and blind people has increased with gradual increase in their population worldwide. The white canes also known as stick system presented in the paper uses artificial intelligence along with various sensors in real time to assist the visually impaired people to help them to navigate their environment independently. The tasks performed by our system are the Image recognition and obstacle detection. The image recognition system consists of a smartphone application utilizing the artificial intelligence. The obstacle detection system consists of ultrasonic sensors to alert the blind or visually impaired of the obstacles in his route [1]. The another project with objective to provide the assistance to the virtually impaired people with the help of smart device using an Android application. This project is an innovative and cause efficient guiding system for the Visually Impaired People (VIP). The major problem for the blinds is to navigate the outdoor environment. Voice control being the one of the main asset for controlling the smart device the system is based on Android application which can be controlled by voice and it is designed for helping the visually impaired to navigate the outdoor environment. The android application helps the user to open any app as and also to make a call to any contact through voice commands. The user commands the mobile device to do something via speech, now these commands are then analyzed immediately by the Speech Recognition Engine (SRE) that converts speech into text for performing out the direct actions [2]. Another system which helps the visually impaired is called as Artificial Vision System for Blind (AVSB), this system consists of ultrasonic sensors and a microcontroller that calculates the distance of the obstacles around the user and to advise him go through an alternative route with the help of an audio feedback. The drawback of this system was that it blocked the daily hearing routine of the visually impaired individuals [3]. According to some tests, the assistant system that uses consists of android application proves to be efficient, cost effective and does not require any special training for the usage of the method. The portable devices most commonly used by everyone have a lot of potential in assisting the visually impaired people on a daily basis. In this project the main objective is to develop an android application for smartphone, specially designed to assist the visually impaired. Dept. of ECE, KLECET, Chikodi

Page 2

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

The application uses inbuilt sensors of a smartphone and also the information received from a few external sensor modules developed for the project. The sensor modules together form an assistive portable system. Communication between smartphone and the sensor modules is made via Bluetooth and Wi-Fi. The communication between smartphone application and its user is made through a text-to-speech conversion module. The assistive activities that are carried out by these modules range from making a phone call, to indoor and outdoor guidance and helping the user to navigate the surroundings independently [4]. The smart assistive system for blinds based on the testing and implementation of the microcontroller (MCU), the device includes a selectable feedback i.e. haptic and audio feedback depending on the user. It this project the smart phone is used to control the device using inbuilt voice commands and Bluetooth. The device is a portable device and it aims at warning the user of the obstacle or object when obstacle is present on the walking path of the blind to avoid the collision. The distance between the user and the obstacle is also calculated with the help of the ultrasonic sensors and the action is performed using ultrasonic echolocation and the data provided by the ultrasonic sensor is now processed by a microcontroller. The microcontroller also controls the feedback system of the device i.e. the audio and haptic feedback mechanism [5]. It is hard to identify people during group meetings and it is drawback for blind people in many situations like professional and educational situations. To overcome by this problem our project used smart phone technology in existence with wireless network to provide audio feedback of the people in front of the blind user. The camera can recognize the face up to a 40degree angle between the direction a person is looking then it scans and give result 96%accurately [6]. Haptic is the science of applying touch sensation and control to interact with computer applications. These gives user sense of touch with computer created natures, so that when virtual things are touched, it looks real and tangible. This technology interfaces with virtual environment via sense of touch by applying force, motions or vibrations to the user. This mechanism is used to create virtual environment, for control of virtual objects, and to enhance remote control of machines and device [7]. Vision is the one of the important part of a human sense and it plays important role in human life. Many papers are published on these topics that propose a variety of computer vision devices and serviced by developing new electronic aids for the blind. In this paper it introduces a Dept. of ECE, KLECET, Chikodi

Page 3

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

system that restores a central function of the visual system which is the identification of surrounding objects. To identify the objects the method is based on the local feature extraction concept. The simulation result using SFIT algorithm and key points matching shows good accuracy of detecting objects [8]. This paper describes machine learning approach for visual object identification which is enable of processing images extremely rapidly and achieving high detection rates. This is differentiated by three key contributions. First one: new image introduction representation is known as “integral image”, which allows the features used our detector to be computed fast. Second one: learning algorithm based on AdaBoost , which selects a small number of critical visual features from a larger set and yield extremely efficient classifiers. Third : the method for combining increasingly more complex classifier in a “cascade “which allows background region of the image to be quickly vanished while spending more computation on promising object like regions [9].

Dept. of ECE, KLECET, Chikodi

Page 4

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 3

OBJECTIVE 1. To help the blind user to analyze the surrounding and identify the objects around him. 2. To help the blind user in identifying human beings using facial recognition. 3. To provide smart assistance to the blind user using Haptic feedback.

Dept. of ECE, KLECET, Chikodi

Page 5

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 4 METHODOLOGY 4.1 Image Recognition and Object detection Image recognition is used to help the blind user to analyze the surrounding and identify the objects around him. And to recognize the people he/she has met earlier, our device uses the tensor flow mechanism for image processing which is captured by the rasp pi camera module as shown in figure 1 below.

Fig.1 Functional Block Diagram for Image Recognition and Object detection Our project includes a Raspberry Pi ("RPI") Zero Wireless, an RPI camera module, haptic feedback motors, server, batteries and an Android application. When the Raspberry Pi is turned on, it waits for a wake command from the user. The wake commands may come from either by saying some wake words into the microphone of a mobile phone, in which our application is running, or by pressing a button which is placed on armband which is fundamental Dept. of ECE, KLECET, Chikodi

Page 6

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

structure of device. The user then verbalizes their command into the phone's microphone, or long-presses the button touch sensor to trigger a default scan of the environment. This command is sent to the server to be relayed to the Raspberry Pi. Once the Raspberry Pi receives the command from the server, it begins to execute the required action. First, the Raspberry Pi captures an image from the camera and converts it into a base64 string to be sent back to the server. The server loads a Tensor Flow instance to describe the scene or pinpoint a specific object.

4.2 Haptic Feedback Mechanism The mechanism of haptic feedback is a complex one. Once the object is detected the haptic motors vibrate in the direction of the object guiding the arm of the user towards the object .

Fig-2: Functional Block Diagram for Haptic Feedback.

Dept. of ECE, KLECET, Chikodi

Page 7

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

If the user simply wants a general description of the image, the server sends its result back to the Android app, in order to explain about the image through mobile speaker. If the user wants to grab a specific object, the process is more complex. The server returns a bounding box of the target object to the Raspberry Pi. The Raspberry pi determines the offset between the object location and the user's hand. Utilizing OpenCV's motion vectors the vibration motors vibrates in the direction that the user's hand must travel to grab the target object, and Raspberry pi pulses once the user has successfully reached the target.

Dept. of ECE, KLECET, Chikodi

Page 8

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 5

FUNCTIONAL BLOCK DIAGRAM We used a Raspberry powered by batteries for the armband, and an Android application for sending voice input to the Pi and for vocalizing the audio feedback. Using a custom armband with embedded cellphone vibration motors(Haptic motors) and an integrated camera, our device could locate objects upon a user’s request, and then guide them to the actual object through directional haptic feedback (if the object is to the left of the user, the motors on the left side of the arm vibrate, etc.) as shown in figure below.

Fig-3: Basic Functional Block Diagram

Dept. of ECE, KLECET, Chikodi

Page 9

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 6

EXPECTED OUTCOMES 

The blind user would be able to navigate the surrounding independently with the help of this smart armband. The user would be able to identify the objects around him and the user will also get the guidance to grab the object through haptic feedback.



The user would be able to identify the person he/she has met earlier through facial recognition.



Our device aims to be the ultimate solution and a smart device that would help the blinds and visually impaired to carry on with the daily routine.

Dept. of ECE, KLECET, Chikodi

Page 10

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 7 ADVANTAGES AND DISADVANTAGES 7.1 ADVANTAGES 

This project helps the user to identify the misplaced objects by object detection through “Haptic feedback” and also assist through Armband to reach the objects.



By this user can able to recognize the person he/she has met earlier from facial recognition.



It will facilitate ease way to recognize the objects and persons.

7.2 DISADVANTAGES 

The smart assistance is provided in app from, a blind user must continuously hold their device out in front of them, an awkward pose to maintain for long periods of time.



Strong Wi-Fi connection is required to operate between the mobile application and processor.



Short range i.e. the assistance is restricted only to the area which the camera module is able to cover.

Dept. of ECE, KLECET, Chikodi

Page 11

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

CHAPTER 8

CONCLUSION In this paper, design and development of a Smart assistive armband through haptic feedback for visually impaired and blind people using Machine learning has been discussed. The system performed three main tasks of Face recognition, object detection and assisting the user through haptic feedback and allowing the user to perform his daily routine independently. Using devices like a smartphone and compact but high quality hardware, our system managed to overcome the hurdle of developing an assistive system which was both efficient and affordable enough for the visually impaired people especially belonging to the middle class families. Future research work involves refining our system so that a more hands free assistive system experience can be provided for the visually disabled people.

Dept. of ECE, KLECET, Chikodi

Page 12

SMART ASSISTIVE ARMBAND THROUGH HAPTIC FEEDBACK FOR VISUALLY IMPAIRED

REFERENCES 1. Sandesh Chinchole, Samir Patel. “Artificial Intelligence and Sensors Based Assistive System for the Visually Impaired People”. Proceedings of the International Conference on Intelligent Sustainable Systems (ICISS 2017) 2. Kasthuri R, Nivetha B, Nivetha B, Veluchamy M, Sivakumar S, “Smart Device for Visually Impaired People” 2017 Third International Conference on Science Technology Engineering & Management (ICONSTEM) 3. A. Iqbal, U. Farooq, H. Mahmood, and M.U. Asad, “A low cost artificial vision system for visually impaired people,” 2009 Second International Conference on Computer and Electrical Engineering, pp. 474-479, December 2009. 4. Laviniu epelea, Ioan Gavrilu, Alexandru Gacsádi “Smartphone Application to Assist Visually Impaired People” 2017 14th International Conference on Engineering of Modern Electric Systems (EMES) 5. D. Munteanu R. Ionel “Voice-Controlled Smart Assistive Device for Visually Impaired Individuals” 12th IEEE international symposium on Electronics and Telecommunication 2016. 6. K.M. Kramer ; D.S. Hedin ; D.J. Rolkosky.”Smartphone based face recognition tool for the blind”, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology 7. B. Divya Jyothi, R.V. Krishnaiah.” Haptic Technology - A Sense of Touch”, International Journal of Science and Research (IJSR), India Online ISSN: 2319-7064 September 2013. 8. Hanen Jabnoun ; Faouzi Benzarti ; Hamid Amiri.” Object detection and identification for blind people in video scene”, 2015 15th International Conference on Intelligent Systems Design and Applications (ISDA) 2015. 9. P. Viola ; M. Jones,” Rapid object detection using a boosted cascade of simple features”, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. 10. www.hackster.io/team-blindsight/blindsight-virtual-eyes-through-haptic-feedback-1f0a8f 11. devpost.com/software/blindsight-virtual-eyes-through-haptic-feedback Dept. of ECE, KLECET, Chikodi

Page 13