Study of Deep Learning-based Hand Gesture Recognition Toward the Design of a Low-cost Prosthetic Hand

¹ 1The University of Aizu, School of Computer Science and Engineering, Adaptive Systems Laboratory, Japan

^* Corresponding author: SAIKAWA Yamatoe-mail: s1260072@u-aizu.ac.jp

Abstract

Background: The surface EMG (sEMG) signal is inherently noisy and, therefore, not a robust input source for prosthetic systems, especially for fatigue, electrode displacement, and sweat conditions. We propose to address these issues by designing a multi-modal approach that combines vision and EMG empowered with appropriate dataset collection.

Methods: In Frame-based, the machine learning model used for recognition was a 2D-CNN. The data is image data that is input to the model by preparing videos showing 10 patterns of hand gestures along with multiple backgrounds, and dividing these videos into frames. These image data are then pre-processed and input to the machine learning model. The model is then evaluated in terms of the accuracy of hand gesture identification using the test data and the loss value, which represents the error between the expected data and the correct data output. In EMG, the Myo armband is placed on the forearm and the sEMG of 200 (Hz) is measured. There are six patterns of hand gestures in this process. Similar to the images, these sEMG data are preprocessed and input to a machine learning model for classification. The model is evaluated the model by the accuracy of hand gesture identification using the test data and the loss value, precision, recall , F1-score.

Results: The value of the loss function in case of frame-based was 0.0770 and the accuracy was 0.9739 at 1000 epochs of the training data. And the value of the loss function values in the test data were 0.1011 for the loss value and 0.9657 for the accuracy. In the case of EMG, the loss value was 0.931 when the time to maintain the gesture was the longest, and the loss value was 0.171. However, Precision, Recall, and F1-score were not the highest at the longest time for some gestures.

Conclusion: In this paper, we created a hand gesture identification software using Frame-based and sEMG, and measured its accuracy and loss value. For sEMG, we used Precision, Recall, and F1-score to check the metrics of each gesture identification. The frame-based results showed good results in both precision and loss values. sEMG showed an improvement in precision and loss values as the time length increased, but there was a tendency to decrease in some indices. In the future, it is necessary to explore the local relationship between finger and forearm to optimize out learning model.