Performance Evaluation of Feature Fusion Approaches for Indian Sign Language Recognition System

B V Poornima; S Srinath; S Rashmi; R Rakshitha

doi:10.17485/IJST/v16i41.1767

Article

Performance Evaluation of Feature Fusion Approaches for Indian Sign Language Recognition System

VIEWS 237
PDF 56

Indian Journal of Science and Technology

DOI: 10.17485/IJST/v16i41.1767

Year: 2023, Volume: 16, Issue: 41, Pages: 3691-3703

Original Article

Performance Evaluation of Feature Fusion Approaches for Indian Sign Language Recognition System

B V Poornima^1*, S Srinath¹, S Rashmi¹, R Rakshitha¹

¹Department of CSE, SJCE, JSSSTU, Mysore, Karnataka

*Corresponding Author
Email: [email protected]

Received Date:14 July 2023, Accepted Date:12 September 2023, Published Date:12 November 2023

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Objectives: To recognize and analyze the Indian sign language (ISL) gestures in simple background using various features and Machine learning classifiers. Methods: a) Data pre-processing: Contour matching approach for hand region segmentation b) Feature extraction: Local features like gradient & key point descriptors were extracted using HOG, SIFT, SURF, LBP, FAST, feature fusion is done by concatenating features of HOG with LBP, SIFT with FAST, BOVW model with SURF. c) Model development: SVM, Random Forest, Logistic regression, Naïve Bayes were trained on large dataset and was experimented with hyper parameter tuning. The experiment was performed on 2 standard datasets which consist of alpha numerals (A-Z & 1-9) in simple black background. Findings: Our research demonstrates that our model consistently achieved a remarkable 100% accuracy rate when utilizing feature fusion techniques on both of the datasets employed in this study. The research findings underscore the need to consider a more comprehensive approach for gesture recognition. Relying solely on distinct features extracted from a single algorithm is shown to be insufficient in addressing the challenges posed by the diverse nature of sign shapes, varying illumination conditions, and different orientations. This emphasizes the importance of exploring hybrid or multi-algorithmic strategies to enhance the accuracy and robustness of gesture recognition systems. Novelty: This research introduces a novel perspective on ISL gesture recognition by emphasizing gestures against a simple black background. The innovative application of feature fusion, combining various elements like hand shape and orientation enhances accuracy. The inclusion of hand segmentation using contour matching algorithm and experimentation on benchmark data adds another layer of novelty, highlighting practical applicability. The experimental results show that SVM has given better results when used with different combinations of feature extractors.

Keywords: ISL, Feature Fusion, Keypoint Descriptors, HOG, SIFT, SURF, LBP, FAST

References

Bhattacharya A, Zope V, Kumbhar K, Borwankar P, Mendes A. Classification of Sign Language Gestures using Machine Learning. International Journal of Advanced Research in Computer and Engineering. 2020;8(12):97–103. Available from: https://doi.org/10.17148/IJARCCE.2019.81219
Kumbhar S, Landge A, Kulkarni A, Solanki D, Kurtadikar V, Karad V. Indian Sign Language Recognition System. International Journal of Innovative Science and Research Technology. 2021;6(6). Available from: https://ijisrt.com/assets/upload/files/IJISRT21JUN1179.pdf
Mariappan HM, Gomathi V. Real-Time Recognition of Indian Sign Language. 2019 International Conference on Computational Intelligence in Data Science (ICCIDS). 2019. Available from: https://doi.org/10.1109/ICCIDS.2019.8862125
Subramanian B, Olimov B, Naik SM, Kim S, Park KHH, Kim J. An integrated mediapipe-optimized GRU model for Indian sign language recognition. Scientific Reports. 2022;12(1):1–16. Available from: https://doi.org/10.1038/s41598-022-15998-7
Pala G, Jethwani JB, Kumbhar SS, Patil SD. Machine Learning-based Hand Sign Recognition. 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS). 2021;p. 356–363. Available from: https://doi.org/10.1109/ICAIS50930.2021.9396030
Sharma A, Mittal A, Singh S, Awatramani V. Hand Gesture Recognition using Image Processing and Feature Extraction Techniques. Procedia Computer Science. 2020;173:181–190. Available from: https://doi.org/10.1016/J.PROCS.2020.06.022
Mapari RB, Kharat GU. Indian Sign Language Alpha-Numeric Character Classification using Neural Network. International Journal of Latest Research Engineering and Technology. 2022. Available from: https://www.academia.edu/28423478/Indian_Sign_Language_Alpha_Numeric_Character_Classification_using_Neural_Network
Dhivyasri S, Hari KB, Akash M, Sona M, Divyapriya S, Krishnaveni V. An efficient approach for interpretation of Indian sign language using machine learning. 2021 3rd International Conference on Signal Processing and Communication (ICPSC). 2021;2021:130–133. Available from: https://doi.org/10.1109/ICSPC51351.2021.9451692
Badhe PC, Kulkarni V. Artificial Neural Network based Indian Sign Language Recognition using hand crafted features. 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). 2020;p. 1–6. Available from: https://doi.org/10.1109/ICCCNT49239.2020.9225294
Amrutha K, Prabu P. ML Based Sign Language Recognition System. 2021 International Conference on Innovative Trends in Information Technology (ICITIIT). 2021. Available from: https://doi.org/10.1109/ICITIIT51526.2021.9399594
Katoch S, Singh V, Tiwary US. Indian Sign Language recognition system using SURF with SVM and CNN. Array. 2022;14:100141. Available from: https://doi.org/10.1016/J.ARRAY.2022.100141
Bello RWW, Mohamed ASA, Talib AZ. Contour Extraction of Individual Cattle From an Image Using Enhanced Mask R-CNN Instance Segmentation Method. IEEE Access. 2021;9:56984–57000. Available from: https://doi.org/10.1109/ACCESS.2021.3072636
Munnaluri V, Pandey V, Singh P. Machine Learning based Approach for Indian Sign Language Recognition. 2022 7th International Conference on Communication and Electronics Systems (ICCES). 2022;p. 1128–1132. Available from: https://doi.org/10.1109/ICCES54183.2022.9835908
Sreemathy R, Turuk M, Kulkarni I, Khurana S. Sign language recognition using artificial intelligence. Education and Information Technologies. 2023;28(5):5259–5278. Available from: https://doi.org/10.1007/s10639-022-11391-z
Savant R, Nasriwala J, Bhatt P. Static Gesture Recognition for Indian Sign Language Alphabets and Numbers using SVM with ORB Keypoints and Image Pixel As Feature. 2023. Available from: https://www.ijert.org/static-gesture-recognition-for-indian-sign-language-alphabets-and-numbers-using-svm-with-orb-keypoints-and-image-pixel-as-feature
Nguyen HBD, Do HN. Deep Learning for American Sign Language Fingerspelling Recognition System. 2019 26th International Conference on Telecommunications (ICT). 2019;p. 314–318. Available from: https://doi.org/10.1109/ICT.2019.8798856
Arulkumar V, Prakash SJ, Subramanian EK, Thangadurai N. An Intelligent Face Detection by Corner Detection using Special Morphological Masking System and Fast Algorithm. 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC). 2021;p. 1556–1561. Available from: https://doi.org/10.1109/ICT.2019.8798856
Dhivyasri S, Hari KB, Akash M, Sona M, Divyapriya S, Krishnaveni V. An efficient approach for interpretation of Indian sign language using machine learning. 2021 3rd International Conference on Signal Processing and Communication (ICPSC). 2021;2021:130–133. Available from: https://doi.org/10.1109/ICSPC51351.2021.9451692
Shravani K, Lakshmi A, Geethika S, Sapna B. Indian Sign Language Character Recognition. IOSR Journal of Computer Engineering. 2020;22(3):14–19. Available from: https://www.iosrjournals.org/iosr-jce/papers/Vol22-issue3/Series-1/B2203011419.pdf
Manikandan J, Krishna BV, Narayan SS, Surendar K. Sign Language Recognition using Machine Learning. 2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES). 2022;p. 3235–3240. Available from: https://doi.org/10.1109/ICSES55317.2022.9914155
Sahoo AK. Indian Sign Language Recognition Using Machine Learning Techniques. Macromolecular Symposia. 2021;397(1). Available from: https://doi.org/10.1002/masy.202000241
Joshi G, Singh S, Vig R. Taguchi-TOPSIS based HOG parameter selection for complex background sign language recognition. Journal of Visual Communication and Image Representation. 2020;71:102834. Available from: https://doi.org/10.1016/j.jvcir.2020.102834
Venugopalan A, Reghunadhan R. Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists. Expert Systems with Applications. 2021;185:115601. Available from: https://doi.org/10.1016/j.eswa.2021.115601

Copyright

© 2023 Poornima et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)