Real-time video based emotion recognition using convolutional neural network and transfer learning

J Sujanaa; S Palanivel

doi:10.17485/IJST/v13i31.1118

Article

Real-time video based emotion recognition using convolutional neural network and transfer learning

VIEWS 3579
PDF 498

Indian Journal of Science and Technology

DOI: 10.17485/IJST/v13i31.1118

Year: 2020, Volume: 13, Issue: 31, Pages: 3222-3229

Original Article

Real-time video based emotion recognition using convolutional neural network and transfer learning

J Sujanaa^1*, S Palanivel²

¹Research Scholar, Department of Computer Science and Engineering, Annamalai University, Annamalainagar, 608002, Tamil Nadu, India. Tel.: +91-7010181616
²Professor, Department of Computer Science and Engineering, Annamalai University,Annamalainagar, 608002, Tamil Nadu, India

*Corresponding Author
Tel: +91-7010181616
Email: [email protected]

Received Date:11 July 2020, Accepted Date:15 August 2020, Published Date:31 August 2020

This work is licensed under a Creative Commons Attribution 4.0 International License.

Abstract

Background/Objectives: The deep learning approaches have paved their way to construct various artificial intelligence products and the proposed system uses a convolutional neural network for detecting real-time emotions of mankind. The objective of the study is to develop a real-time application for emotion recognition using convolutional neural network and transfer learning methods. Methods/Statistical analysis: The proposed system considers happy, normal and surprised categories of emotions. The system consists of four major steps: dataset collection, training, validation, and real-time testing. The dataset is comprised of face images containing emotions such as happy,normal and surprised in the form of video frames. The face and mouth regions are detected using the Haar-Based Cascade classifier at 20 frames per second. Findings: The convolutional neural network (CNN) is trained using mouth images and the pre-trained models VGG16 and VGG19 are trained with face images. The trained model is used to detect the emotions in the live webcam video. The experimental results show that the CNN model trained using mouth images gives an accuracy of 85.71% and the pre-trained models trained with face images using transfer learning method achieves an accuracy of 77.78%. The proposed system using CNN outperforms the pre-trained models for recognizing the emotions in real-time video. Novelty/Applications:The proposed system is entirely based on the mouth region video frames and the real-time emotion recognition system is developed. This work can detect the three emotions in an unconstrained laboratory environment.

Keywords: Convolutional neural network; mouth detection; pre-trained models; real-time emotion recognition; transfer learning

References

De A, Saha A. A comparative study on different approaches of real time human emotion recognition based on facial expression detection. In: Conference Proceeding - 2015 International Conference on Advances in Computer Engineering and Applications, ICACEA . p. 483–487.
Ozdemir MA, Elagoz B, Alaybeyoglu A, Sadighzadeh R, Akan A. Real time emotion recognition from facial expressions using CNN architecture. TIPTEKNO 2019 - Tip Teknol Kongresi. 2019;p. 1–4.
Sokolov D, Patkin M. Real-time emotion recognition on mobile devices. In: Proc - 13th IEEE Int Conf Autom Face Gesture Recognition. (Vol. 787) 2018.
Majumdar A, Ward RK. Discriminative sift features for face recognition. Canadian Conference on Electrical and Computer Engineering. 2009;p. 27–30.
Happy SL, Routray A. Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing. 2015;6(1):1–12. Available from: https://dx.doi.org/10.1109/taffc.2014.2386334
Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M. Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters. 2018;115:101–106. Available from: https://dx.doi.org/10.1016/j.patrec.2018.04.010
Kaya H, Gürpınar F, Salah AA. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image and Vision Computing. 2017;65:66–75. Available from: https://dx.doi.org/10.1016/j.imavis.2017.01.012
Anusha AV, Jayasree JK, Bhaskar A, Aneesh RP. Facial expression recognition and gender classification using facial patches. In: 2016 Int Conf Commun Syst Networks, ComNet. p. 200–204.
Liang Z, Oba S, Ishii S. An unsupervised EEG decoding system for human emotion recognition. Neural Networks. 2019;116:257–268. Available from: https://dx.doi.org/10.1016/j.neunet.2019.04.003
Bhavan A, Chauhan P, Hitkul, Shah RR. Bagged support vector machines for emotion recognition from speech. . Knowledge-Based Syst. 2019. Available from: https://doi.org/10.1016/j.knosys.2019.104886
Hassan MM, Alam MGR, Uddin MZ, Huda S, Almogren A, Fortino G. Human emotion recognition using deep belief network architecture. Information Fusion. 2019;51:10–18. Available from: https://dx.doi.org/10.1016/j.inffus.2018.10.009
Nalini NJ, Palanivel S. Music emotion recognition: The combined evidence of MFCC and residual phase. Egyptian Informatics Journal. 2016;17(1):1–10. Available from: https://dx.doi.org/10.1016/j.eij.2015.05.004
Karpathy A, Leung T. Large-scale Video Classification with Convolution Neural Networks. In: Proceedings of International Computer Vision and Pattern Recognition. (pp. 10-20) 2018.
Jaderberg M, Simonyan K, Vedaldi A, Zisserman A. Reading Text in the Wild with Convolutional Neural Networks. International Journal of Computer Vision. 2016;116:1–20. Available from: https://dx.doi.org/10.1007/s11263-015-0823-z
Jaderberg M, Simonyan K, Vedaldi A, Zisserman A. Reading Text in the Wild with Convolutional Neural Networks. International Journal of Computer Vision. 2016;116:1–20. Available from: https://dx.doi.org/10.1007/s11263-015-0823-z
Wu H, Feng J, Tian X, Sun E, Liu Y, Dong B. EMO: Real-time emotion recognition from single-eye images for resource-constrained eyewear devices. In: MobiSys 2020 - Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. (pp. 448-61) Association for Computing Machinery, Inc..
Lech M, Stolar M, Best C, Bolia R. Real-Time Speech Emotion Recognition Using a Pre-trained Image Classification Network: Effects of Bandwidth Reduction and Companding. Frontiers in Computer Science. 2020;2. Available from: https://dx.doi.org/10.3389/fcomp.2020.00014
Keras Library. Available from: https://keras.io/
Abadi M. Large-Scale Learning on Heterogeneous Distributed Systems. Prelim White Pap. 2016;1(212):1–19. Available from: http://download.tensorflow.org/paper/whitepaper2015.pdf
Viola P, MJ. Rapid Object Detection using a Boosted Cascade of Simple Features. . Comput Vis Patter Recognit. 2001;394(1-3):1–9. Available from: https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf
Pedregosa FF, Michel V, Grisel OO, Blondel M, Prettenhofer P, Weiss R. Scikit-learn: Machine Learning in Python. J Mach Learn Res [Internet]. 2011;12:2825–2855. Available from: http://scikit-learn.sourceforge.net
ImageNet Overview . (accessed 2019-09-21) Available from: https://en.wikipedia.org/wiki/ImageNet#ImageNet_Challenge
Bisong E, Bisong E. Google Colaboratory. In: Building Machine Learning and Deep Learning Models on Google Cloud Platform. 2019.
Ganesan K. Text Mining, Analytics & More: Computing Precision and Recall for Multi-Class Classification Problems. (accessed 2019-09-29) Available from: http://text-analytics101.rxnlp.com/2014/10/computing-precision-and-recall-for.html

Copyright

© 2020 Sujanaa & Palanivel.This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee).