• P-ISSN 0974-6846 E-ISSN 0974-5645

Indian Journal of Science and Technology


Indian Journal of Science and Technology

Year: 2023, Volume: 16, Issue: 45, Pages: 4177-4185

Original Article

Transforming Sign Language into Text and Speech through Deep Learning Technologies

Received Date:11 October 2023, Accepted Date:30 October 2023, Published Date:05 December 2023


Objective: The goal of the proposed work is to leverage deep learning technologies to create an efficient and accurate system for transforming sign language into text and speech. People deliver their ideas, feelings, and experiences to others around them via their interactions with each other. The hand gesture plays a significant role since it reflects the user's thoughts more rapidly than other motions (head, face, eye, and body). For deaf-mute people with disabilities, this is still not the case. Sign language facilitates communication among deaf-mute individuals. An individual who is deaf-mute can communicate without the use of acoustic noises by using sign language. Methods: Convolutional neural networks (CNNs) are generally used to recognize and extract characteristics from sign language motions. These neural networks are employed to recognize and extract critical features from sign language gestures. These features are processed by natural language processing models for textual translation. Finally, neural text-to-speech (TTS) technology is used to translate the textual translations into synthesized speech, thereby bridging the communication gap for the Deaf community. To establish an inclusive and accessible communication system, this technique combines computer vision, natural language processing, and speech synthesis. Findings: The datasets used in this technique include hand gesture images, which contain different hand poses and expressions. It is used to train and assess the model. The experiment findings show an accuracy of 97.6% with a precision of 94.1%, a recall of 96.8%, and an F1-score of 95.9%. Novelty: This approach displays a cogent translation from text to speech and achieves an outstanding translation accuracy of 97.6% from sign language to text, producing a natural and understandable output.

Keywords: Sign Language Translation, Deep Learning, Convolutional Neural Networks, Sequence­To­Sequence Models, Attention Mechanisms, Neural Text­To­Speech


  1. Buttar AM, Ahmad U, Gumaei AH, Assiri A, Akbar MA, Alkhamees BF. Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs. Mathematics. 2023;11(17):3729. Available from: https://doi.org/10.3390/math11173729
  2. Abraham E, Nayak A, Iqbal A. Real-Time Translation of Indian Sign Language using LSTM. 2019 Global Conference for Advancement in Technology (GCAT). 2019;p. 1–5. Available from: https://doi.org/10.1109/GCAT47503.2019.8978343
  3. Uyyala P. Sign Language Recognition Using Convolutional Neural Networks. Journal of interdisciplinary cycle research. 2022;14:1198–1207. Available from: https://doi.org/10.17613/47ga-zw60
  4. Sreenath S, Daniels DI, Ganesh ASD, Kuruganti YS, Chittawadigi RG. Monocular Tracking of Human Hand on a Smart Phone Camera using MediaPipe and its Application in Robotics. 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC). 2021. Available from: https://doi.org/10.1109/R10-HTC53172.2021.9641542
  5. Bohra T, Sompura S, Parekh K, Raut P. Real-Time Two Way Communication System for Speech and Hearing Impaired Using Computer Vision and Deep Learning. 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT). 2019. Available from: https://doi.org/10.1109/ICSSIT46314.2019.8987908
  6. Elboushaki A, Hannane R, Afdel K, Koutti L. MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Systems with Applications. 2020;139:112829. Available from: https://doi.org/10.1016/j.eswa.2019.112829
  7. Liu JE, An FP. Image Classification Algorithm Based on Deep Learning-Kernel Function. Scientific Programming. 2020;p. 1–14. Available from: https://doi.org/10.1155/2020/7607612
  8. Oudah M, Al-Naji A, Chahl J. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. Journal of Imaging. 2020;6(8):73. Available from: https://doi.org/10.3390/jimaging6080073
  9. Al-Hammadi M, Muhammad G, Abdul W, Alsulaiman M, Hossain MS. Hand Gesture Recognition Using 3D-CNN Model. IEEE Consumer Electronics Magazine. 2020;9(1):95–101. Available from: https://doi.org/10.1109/MCE.2019.2941464
  10. Vaidya O, Gandhe S, Sharma A, Bhate A, Bhosale V, Mahale R. Design and Development of Hand Gesture based Communication Device for Deaf and Mute People. 2020 IEEE Bombay Section Signature Conference (IBSSC). 2020. Available from: https://doi.org/10.1109/IBSSC51096.2020.9332208
  11. Li Z, Liu F, Yang W, Peng S, Zhou J. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Transactions on Neural Networks and Learning Systems. 2022;33(12):6999–7019. Available from: https://doi.org/10.1109/TNNLS.2021.3084827
  12. Smedt D, Quentin H, Wannous JP, Vandeborre. Heterogeneous hand gesture recognition using 3D dynamic skeletal data. Computer Vision and Image Understanding. 2019;181:60–72. Available from: https://doi.org/10.1016/j.cviu.2019.01.008
  13. Sharma A, Mittal A, Singh S, Awatramani V. Hand Gesture Recognition using Image Processing and Feature Extraction Techniques. Procedia Computer Science. 2020;173:181–190. Available from: https://doi.org/10.1016/j.procs.2020.06.022
  14. Jie HJ, Wanda P. RunPool: A Dynamic Pooling Layer for Convolution Neural Network. International Journal of Computational Intelligence Systems. 2020;13(1):66. Available from: https://doi.org/10.2991/ijcis.d.200120.002
  15. Shokat S, Riaz R, Rizvi SS, Khan K, Riaz F, Kwon SJ. Analysis and Evaluation of Braille to Text Conversion Methods. Mobile Information Systems. 2020;p. 1–14. Available from: https://doi.org/10.1155/2020/3461651
  16. Rahaman MA, Hossain MDP, Rana MM, Rahman MDA, Akter T. A Rule Based System for Bangla Voice and Text to Bangla Sign Language Interpretation. 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI). 2020. Available from: https://doi.org/10.1109/STI50764.2020.9350468
  17. RS, Hegde SR, Chinmaya K, Priyesh A, Manjunath AS, Arunakumari BN. Indian Sign Language to Speech Conversion Using Convolutional Neural Network. In: 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon). (pp. 1-5).


© 2023 Duraisamy et al.  This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)


Subscribe now for latest articles and news.