Transforming Sign Language into Text and Speech through Deep Learning Technologies

Premkumar Duraisamy; A Abinayasrijanani; M Amrit Candida; P Dinesh Babu

doi:10.17485/IJST/v16i45.2583

Article

Transforming Sign Language into Text and Speech through Deep Learning Technologies

VIEWS 316
PDF 67

Indian Journal of Science and Technology

DOI: 10.17485/IJST/v16i45.2583

Year: 2023, Volume: 16, Issue: 45, Pages: 4177-4185

Original Article

Transforming Sign Language into Text and Speech through Deep Learning Technologies

Premkumar Duraisamy^1*, A Abinayasrijanani², M Amrit Candida², P Dinesh Babu²

¹Assistant Professor (Sr. G.), Department of Computer Science and Engineering, KPR Institute of Engineering and Technology, Coimbatore, Tamil Nadu, India
²B.E Student, Department of Computer Science and Engineering, KPR Institute of Engineering and Technology, Coimbatore, Tamil Nadu, India

*Corresponding Author
Email: [email protected]

Received Date:11 October 2023, Accepted Date:30 October 2023, Published Date:05 December 2023

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Objective: The goal of the proposed work is to leverage deep learning technologies to create an efficient and accurate system for transforming sign language into text and speech. People deliver their ideas, feelings, and experiences to others around them via their interactions with each other. The hand gesture plays a significant role since it reflects the user's thoughts more rapidly than other motions (head, face, eye, and body). For deaf-mute people with disabilities, this is still not the case. Sign language facilitates communication among deaf-mute individuals. An individual who is deaf-mute can communicate without the use of acoustic noises by using sign language. Methods: Convolutional neural networks (CNNs) are generally used to recognize and extract characteristics from sign language motions. These neural networks are employed to recognize and extract critical features from sign language gestures. These features are processed by natural language processing models for textual translation. Finally, neural text-to-speech (TTS) technology is used to translate the textual translations into synthesized speech, thereby bridging the communication gap for the Deaf community. To establish an inclusive and accessible communication system, this technique combines computer vision, natural language processing, and speech synthesis. Findings: The datasets used in this technique include hand gesture images, which contain different hand poses and expressions. It is used to train and assess the model. The experiment findings show an accuracy of 97.6% with a precision of 94.1%, a recall of 96.8%, and an F1-score of 95.9%. Novelty: This approach displays a cogent translation from text to speech and achieves an outstanding translation accuracy of 97.6% from sign language to text, producing a natural and understandable output.

Keywords: Sign Language Translation, Deep Learning, Convolutional Neural Networks, SequenceToSequence Models, Attention Mechanisms, Neural TextToSpeech

References

Buttar AM, Ahmad U, Gumaei AH, Assiri A, Akbar MA, Alkhamees BF. Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs. Mathematics. 2023;11(17):3729. Available from: https://doi.org/10.3390/math11173729
Duraisamy P, Natarajan Y, Jeya IJS, Niranjani V. Sensor Automation for Industrial Applications and Prediction of Energy Level for Home Appliances Using Machine Learning Algorithm. In: 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS). (pp. 1648-1654) IEEE. 2023.
Abraham E, Nayak A, Iqbal A. Real-Time Translation of Indian Sign Language using LSTM. 2019 Global Conference for Advancement in Technology (GCAT). 2019;p. 1–5. Available from: https://doi.org/10.1109/GCAT47503.2019.8978343
Uyyala P. Sign Language Recognition Using Convolutional Neural Networks. Journal of interdisciplinary cycle research. 2022;14:1198–1207. Available from: https://doi.org/10.17613/47ga-zw60
Sreenath S, Daniels DI, Ganesh ASD, Kuruganti YS, Chittawadigi RG. Monocular Tracking of Human Hand on a Smart Phone Camera using MediaPipe and its Application in Robotics. 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC). 2021. Available from: https://doi.org/10.1109/R10-HTC53172.2021.9641542
Bohra T, Sompura S, Parekh K, Raut P. Real-Time Two Way Communication System for Speech and Hearing Impaired Using Computer Vision and Deep Learning. 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT). 2019. Available from: https://doi.org/10.1109/ICSSIT46314.2019.8987908
Elboushaki A, Hannane R, Afdel K, Koutti L. MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Systems with Applications. 2020;139:112829. Available from: https://doi.org/10.1016/j.eswa.2019.112829
Liu JE, An FP. Image Classification Algorithm Based on Deep Learning-Kernel Function. Scientific Programming. 2020;p. 1–14. Available from: https://doi.org/10.1155/2020/7607612
Oudah M, Al-Naji A, Chahl J. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. Journal of Imaging. 2020;6(8):73. Available from: https://doi.org/10.3390/jimaging6080073
Al-Hammadi M, Muhammad G, Abdul W, Alsulaiman M, Hossain MS. Hand Gesture Recognition Using 3D-CNN Model. IEEE Consumer Electronics Magazine. 2020;9(1):95–101. Available from: https://doi.org/10.1109/MCE.2019.2941464
Vaidya O, Gandhe S, Sharma A, Bhate A, Bhosale V, Mahale R. Design and Development of Hand Gesture based Communication Device for Deaf and Mute People. 2020 IEEE Bombay Section Signature Conference (IBSSC). 2020. Available from: https://doi.org/10.1109/IBSSC51096.2020.9332208
Li Z, Liu F, Yang W, Peng S, Zhou J. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Transactions on Neural Networks and Learning Systems. 2022;33(12):6999–7019. Available from: https://doi.org/10.1109/TNNLS.2021.3084827
Smedt D, Quentin H, Wannous JP, Vandeborre. Heterogeneous hand gesture recognition using 3D dynamic skeletal data. Computer Vision and Image Understanding. 2019;181:60–72. Available from: https://doi.org/10.1016/j.cviu.2019.01.008
Sharma A, Mittal A, Singh S, Awatramani V. Hand Gesture Recognition using Image Processing and Feature Extraction Techniques. Procedia Computer Science. 2020;173:181–190. Available from: https://doi.org/10.1016/j.procs.2020.06.022
Duraisamy P, Natarajan Y, Niranjani V, Parvathy K. Optimized Detection of Continuous Gravitational-Wave Signals using Convolutional Neural Network. In: 2023 3rd International conference on Artificial Intelligence and Signal Processing (AISP). (pp. 1-5) IEEE. 2023.
Jie HJ, Wanda P. RunPool: A Dynamic Pooling Layer for Convolution Neural Network. International Journal of Computational Intelligence Systems. 2020;13(1):66. Available from: https://doi.org/10.2991/ijcis.d.200120.002
Shokat S, Riaz R, Rizvi SS, Khan K, Riaz F, Kwon SJ. Analysis and Evaluation of Braille to Text Conversion Methods. Mobile Information Systems. 2020;p. 1–14. Available from: https://doi.org/10.1155/2020/3461651
Rahaman MA, Hossain MDP, Rana MM, Rahman MDA, Akter T. A Rule Based System for Bangla Voice and Text to Bangla Sign Language Interpretation. 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI). 2020. Available from: https://doi.org/10.1109/STI50764.2020.9350468
Duraisamy P, Natarajan Y, Preethaa KRS, Mouthami K. Sentiment Analysis on Drug Reviews Using Diverse Classification Techniques. In: 2022 3rd International Conference on Communication, Computing and Industry 4.0 (C2I4). (pp. 1-5) IEEE. 2022.
RS, Hegde SR, Chinmaya K, Priyesh A, Manjunath AS, Arunakumari BN. Indian Sign Language to Speech Conversion Using Convolutional Neural Network. In: 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon). (pp. 1-5).
Qin W, Mei X, Chen Y, Zhang Q, Yao Y, Hu S. Sign Language Recognition and Translation Method based on VTN. In: 2021 International Conference on Digital Society and Intelligent Systems (DSInS). (pp. 111-115) IEEE. 2021.
Shokoori AF, Shinwari M, Popal JA, Meena J. Sign Language Recognition and Translation into Pashto Language Alphabets. In: 2022 6th International Conference on Computing Methodologies and Communication (ICCMC). (pp. 1401-1405) IEEE. 2022.
Duraisamy P, Yuvaraj S, Natarajan Y, Niranjani V. An Overview of Different Types of Recommendations Systems - A Survey. In: 2023 4th International Conference on Innovative Trends in Information Technology (ICITIIT). (pp. 1-5) IEEE. 2023.
Kim CJ, Park HM. Per-frame Sign Language Gloss Recognition. In: 2021 International Conference on Information and Communication Technology Convergence (ICTC). (pp. 1125-1127) IEEE. 2021.
Datta PK, Biswas A, Ghosh A, Chaudhury N. Creation of Image Segmentation Classifiers for Sign Language Processing for Deaf and Dumb. In: 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO). (pp. 772-775) IEEE. 2020.

Copyright

© 2023 Duraisamy et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)