International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 05 Issue: 12 | Dec 2018 p-ISSN: 2395-0072 www.irjet.net A Survey on Control of Mechanical Arm based on Hand Gesture Recognition using Wearable IMU Sensors Sherin Samuel1, Ramandra Yadav2, Omkar Jadhav3, Sumant Ranmale4, Mahesh S. Shinde5 1,2,3,4Student, Dept. of Computer Engineering, MES College of Engineering, Pune, Maharashtra, India. Professor, Dept. of Computer Engineering, MES College of Engineering, Pune, Maharashtra, India. ---------------------------------------------------------------------***---------------------------------------------------------------------1.1 Gesture Classification Abstract - Gesture has played a crucial role in reducing 5Assistant the gap between human and machine. Hand gesture recognition is applied in many technologies like games, wireless devices, mobile application, etc. In this paper, we have proposed a system in which the mechanical arm imitates the gesture performed by the operator using a wearable and machine learning algorithm for classification of gestures. The wearable is based on Inertial Measurement Unit (IMU) sensors which is gaining significant importance in field of Gesture Recognition. The algorithm used for gesture classification is Support Vector Machine which can recognize movement with less computation time and higher accuracy, and can also model non-linear relation with more precise classification using SVM kernels. A lot of work has been done in the field of human face expression recognition which is an active research topic since the early nineties. There has been a lot of advances in the past few years for face detection and tracking while the problem of recognizing hand gestures is under explored. The reason is the higher level of complexity of hand gestures as compared to face detection. Hand gestures can be recorded using devices like depth cameras or data gloves equipped with sensors. Use of cameras and image processing introduces complex computations which may be difficult to overcome hence gloves equipped with sensors are ideally suited for this task. There are a lot of fields of work like nuclear power plants, underwater exploration, disaster prone areas or other places where human intervention is difficult or there are situations where the machine operator has to control machine through long distance. Traditional methods like remote control would be tough to handle in those conditions. In such situations gesture based control of the machine helps in handling it easily where machines like mechanical arm would imitate the gesture of the operator and the work could be performed from a distance. Key Words: Gesture Classification, IMU sensors, Wearable device, Mechanical arm, Human-machine Interaction. 1. INTRODUCTION Human-Machine Interaction (HMI) is a widespread field based on communication between human and machine. HMI is the interaction between human and machine which is done by analyzing human behavior and appearance various methods are proposed in human machine interaction. Some of them are through voice, keyboard, camera, etc. Gesture is one of the methods used for HMI which is a non-verbal communication method based on body movements. Briefly, in this paper we have proposed a system in which the mechanical arm imitates the gesture of the operator. The gesture data is collected through an Inertial Measurement Unit (IMU) which consists of a 3-axis gyroscope and accelerometer. A virtual reality head gear helps in viewing the workspace through a camera situated there. Among popular classification algorithms it is found out that Support Vector Machine (SVM) is proven to recognize gestures in less computation time with higher accuracy rate. The input to classifier is the data from IMU sensors and its output is the corresponding label which defines the action performed by the mechanical arm. The wearable and mechanical arm are connected to a computer wirelessly through Bluetooth. The machine learning algorithm can be run on the computer and the output of the algorithm will be passed to Raspberry Pi that controls mechanical arm. Various sensors are used for gesture recognition. It has been used in various areas like gaming [9], sign language recognition [8], appliance control. Some of the methods are camera based [10], inertial sensor based [6,7], electrical field based [2] or combination of them. The gestures are recognized by various algorithms like support vector machine [1], artificial neural network [3], convolutional neural network [8]. Inertial sensor based gesture recognition is gaining popularity because of its low cost, lightweight, low power design with great precision. Inertial measurement unit (IMU) consists of accelerometer and gyroscope responsible for finding the positioning of an object in 3D space. These sensors would help in finding positioning coordinates of hand movement. © 2018, IRJET | Impact Factor value: 7.211 | ISO 9001:2008 Certified Journal | Page 780 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 05 Issue: 12 | Dec 2018 p-ISSN: 2395-0072 www.irjet.net 2. LITERATURE REVIEW 2.5 Classification of Hand Gestures from Wearable IMU using Deep Neural Network 2.1 Hand Body Language Gesture Recognition Based on Signals From Specialized Glove and Machine Learning Algorithms [5] Karush Suri et al. In this paper an improved approach for gesture classification is proposed using a Deep Neural Network for classification of six hand gestures corresponding to the signs used in the Indian sign language using signals from three IMU units. Principle component analysis (PCA) is used for dimensionality reduction and data is normalized. An accuracy of about 97.386% is observed. [1] Paweł Pławiak et al. This paper presents a system for quick and effective recognition of hand gestures based on data glove equipped with 10 sensors. Collected data was preprocessed using normalization and PCA analysis to test increase in classification sensitivity with reduction of data. They designed models using 3 machine learning algorithms based on Neural Network, Support Vector machine and Knearest neighbors. Through results they found out that SVM gave highest accuracy rate (98.32%) as compared to PNN (97.23%) and KNN (97.36%). 2.6 Flexible Gesture Recognition Using Wearable Inertial Sensors [6] Huda Abualola et al. In this paper a glove based Gesture Recognition system is proposed that tracks hand movement using inertial sensors. The proposed glove is able to recognize American Sign language letters and communicate them for external displays on smart-phones. The algorithm used is based on linear discriminant analysis (LDA) which allows accurate and low complexity classification with improved clustering and reduced dimensionality. 2.2 An Efficient Approach to Recognize Hand Gestures Using Machine-Learning Algorithms [2] Md Ferdous Wahid et al. This paper presents use of Electromyography (EMG) sensors for capturing gesture data. Five machine learning algorithms such as K-nearest neighbor (KNN), Discriminant analysis (DA), Naive Bayes , Random Forest and Support Vector Machine (SVM) were used to classify 3 different hand gestures. SVM showed maximum accuracy (97.56%) in classification which was further improved by normalizing EMG features (98.73). 2.7 Gesture Recognition with Wearable 9-axis Sensors [7] Fang-Ting Liu et al. This paper presents use of SVM and Inertial Measurement units for gesture recognition. For achieving higher classification accuracy feature extraction is applied using principal component analysis and linear discriminant analysis. Principal Component Analysis is an algorithm used for feature extraction which helps in improving gestures classification sensitivity and accuracy considerably. SVM is used with radial basis function (RBF) kernel and a non linear SVM called Gausian SVM which provides a more precise classification and can also add nonlinearity. Kernels help in improving the classification sensitivity and accuracy considerably. 2.3 Artificial Neural Networks for Gesture Classification with Inertial Motion Sensing Armbands [3] Ananta Srisuphab et al. This paper presents the design of an automated tool to assist construction workers in the hand signal communicative channel. This system uses Myo sensors which is a combination of surface Electromyography (EMG) and Inertial Measurement Unit (IMU). The algorithm used for classification is Multi-Layer Neural Network. Daubechies wavelet transforms was used to analyze in frequency domain along with normalization and 10-fold cross validation was performed. The highest mean classification accuracy achieved was 88.176%. 2.8 Training CNN for 3D Sign Language Recognition with color texture coded Joint Angular Displacement Maps [8] E.Kiran Kumar et al. In this paper Convolutional Neural Network (CNN) is used in the recognition of 3D motion-captured sign language. The 3D information of each sign is interpreted using Joint Angular Displacement maps (JADM) which uses both angular measurements and joint distance maps (JDM) and encodes skeletal data into color texture images. 2.4 A Sliding Window Approach to Natural Hand Gesture Recognition using a Custom Data Glove [4] Granit Luzhnica et al. This paper presents a gesture based system that can be adopted for general-purpose control of and communication with computing systems that are currently performed via mouse or smart phone gestures. This system uses a sliding window approach for data processing and hence suitable for stream data processing. A combination of linear discriminant analysis and logistic regression is used which resulted in an accuracy of over 98.5% on a continuous data stream scenario. © 2018, IRJET | Impact Factor value: 7.211 2.9 Hand Gesture Recognition and Real-time Game Control Based on A Wearable Band with 6-axis Sensors [9] Yande Li et al. This paper proposes a system based on hand gesture recognition and real time game control to run | ISO 9001:2008 Certified Journal | Page 781 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 05 Issue: 12 | Dec 2018 p-ISSN: 2395-0072 www.irjet.net an android game named Fly Birds. Accelerometer and gyroscope in wrist band is used to collect hand gesture information and Kinect camera captures video information which is mainly used to mark the data during gesture recognition stage. Sliding window and DTW algorithm is used to detect gestures in real time game control. used to predict the actions to be performed by the mechanical arm. There are many types of algorithms used for classification like Probabilistic Neural Network, Knearest neighbor, Decision trees, Support Vector Machines (SVM), etc. The output of machine learning model is in the form of labels that are mapped to the type of action performed by mechanical arm. The output labels get mapped to the servo motor rotation angles of mechanical arm and accordingly each joint movement takes place. 2.10 Imitation and Learning of Human Hand Gesture Tasks of the 3D Printed Robotic Hand by Using Artificial Neural Networks 3.4 Computer Software [10] Mehmet Celalettin Ergene et al. In this paper it is explained how robot hand can learn via imitation using image processing based on Artificial Neural network. Features were extracted using image processing algorithms such as filtering and background subtraction. The success rate of network is 90.1%. A Linux software is designed that connects the mechanical arm and wearable glove through Bluetooth for the data transfer to take place among them. The sensor data is send to this software that runs the machine learning model and it produces the output which is then transfered to the Raspberry Pi which controls the mechanical arm. 3. PROPOSED SYSTEM 4. CONCLUSIONS In this paper, we developed a system for imitation of hand gestures by a mechanical arm which would help operate machines wirelessly from a distance easily. This provides a very comfortable work environment for operator and the handling of machines with complex controls can be done easily through hand gestures. For wireless communication Bluetooth and Wi-Fi can be used .We have combined algorithms like SVM and PCA to design an effective machine learning model. This system can be further developed to perform computations over cloud and control machines through Internet if bluetooth or Wi-Fi connection is not possible. Figure -1: System Architecture ACKNOWLEDGEMENT 3.1 Gesture Data acquisition We take this opportunity to express our deep sense of gratitude for our guide Prof. Mahesh S. Shinde sir (Assistant Professor, MES College of Engineering,Pune), for providing us this good opportunity to select and present this topic and also for providing all the facilities and guidance required for the successful completion of our research work. The gesture data is collected from a wearable glove mounted with Inertial measurement unit (IMU) sensors. The IMU unit is MPU 6050 which has 3-axis accelerometer and gyroscope. It gives output in the form of x-y-z co-ordinates. 3.2 Preprocessing REFERENCES Data Preprocessing is an important step before applying machine learning algorithms. Preprocessing is done in order to improve quality of data and also helps improve sensitivity of classification through processes like dimensionality reduction and normalization. Dimensionality reduction is performed through Principle Component Analysis (PCA) algorithm which helps in improving classification accuracy considerably. [1] Pawel Plawiak, Toma z o nicki Michal Nied zwiecki Zbi law Tabor and Krzy ztof Rzecki “Hand Body Language Gesture Recognition Based on Signals From pecialized Glove and Machine Learning Algorithm ” IEEE Transactions on Industrial Informatics, 2016. [2] Md Ferdous Wahid, Reza Tafreshi, Mubarak Al-Sowaidi, Reza Langari “An Efficient Approach to Recognize Hand Gestures Using Machine-Learning Algorithm ” IEEE 4th Middle East Conference on Biomedical Engineering (MECBME), 2018. 3.3 Machine Learning Model The model gets prepared after learning process using the data acquired from sensors. This machine learning model is © 2018, IRJET | Impact Factor value: 7.211 | ISO 9001:2008 Certified Journal | Page 782 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 05 Issue: 12 | Dec 2018 p-ISSN: 2395-0072 www.irjet.net [3] Ananta ri uphab Piyanuch ilapachote “Artificial Neural Networks for Gesture Classification with Inertial Motion en ing Armband ” IEEE Region 10 Conference (TENCON), 2016. [4] Granit Luzhnica, Jorg Simon, Elisabeth Lex, Viktoria Pammer “A liding Window Approach to Natural Hand Gesture Recognition using a Cu tom Data Glove” IEEE Symposium on 3D User Interfaces, 19-20 march 2016. [5] Karu h uri Rinki Gupta “Cla ification of Hand Gestures from Wearable IMUs using Deep Neural Network” 2nd International Conference on Inventive Communication and Computational Technologies, 2018. [6] Huda Abualola, Hanin Al Ghothani, Abdulrahim Naser Eddin Nawaf Almoo a Kin Poon “Flexible Ge ture Recognition U ing Wearable Inertial en or ” 59th International Midwest Symposium on Circuits and Systems (MWSCAS), 16-19 October 2016. [7] Fang-Ting Liu, Yong-Ting Wang, Hsi-Pin Ma “Ge ture Recognition with Wearable 9-axi en or ” IEEE ICC SAC Symposium E-Health Track, 2017. [8] E.Kiran Kumar, P.V.V.Kishore, A.S.C.S.Sastry, M.Teja Kiran Kumar D.Anil Kumar “Training CNN for 3D Sign Language Recognition with color texture coded Joint Angular Di placement Map ” IEEE ignal Proce ing Letters, May 2018. [9] Yande Li, Taiqian Wang, Aamir khan, Lian Li, Caihong Li, Yi Yang Li Liu “Hand Ge ture Recognition and Real-time Game Control Based on A Wearable Band with 6-axis en or ” International Joint Conference on Neural Networks (IJCNN), 2018. [10] Mehmet CelalettinErgene, Akif Durdu, Halil Cetin, “Imitation and Learning of Human Hand Ge ture Ta k of the 3D Printed Robotic Hand by Using Artificial Neural Network ” International Conference – 8th Edition Electronics, Computers and Artificial Intelligence 30 June -02 July, 2016. © 2018, IRJET | Impact Factor value: 7.211 | ISO 9001:2008 Certified Journal | Page 783