• P-ISSN 0974-6846 E-ISSN 0974-5645

Indian Journal of Science and Technology


Indian Journal of Science and Technology

Year: 2022, Volume: 15, Issue: 47, Pages: 2619-2627

Original Article

Context Aware Image Sentiment Classification using Deep Learning Techniques

Received Date:20 September 2022, Accepted Date:08 November 2022, Published Date:21 December 2022


Objectives: To propose context aware sentiment classification using deep learning techniques. Methods: We used EfficientNetB-7 deep learning framework for caption generation for the input image and to classify the sentiment of generated caption using machine learning techniques. First, we employ several real-time and synthetic image datasets, then apply pre-processing and normalization for data balancing. Then efficient module implementation for feature extraction and selection using convolutional and pooling layers were done. Despite this proceeding, it generates the caption for respective images. The various feature extraction and selection Natural Language Processing (NLP) techniques such as TF-IDF, lemmas, dependency and correlational features have been used and classify the sentiment label using attention model and greedy approach. Finally, generating the blue score for the entire testing dataset and show the effectiveness of the proposed system. Findings: Our model gives higher accuracy with different deep learning techniques which is demonstrated in result section. The proposed model archives 73.80% average accuracy for EMOTIC dataset. The module has evaluated with different features and deep learning classification algorithms proposed earlier. Novelty: This research is the collaboration of Deep learning and machine learning classification techniques. We first extract the visual features from the input image using deep learning and classify with machine learning with the collaboration of NLP processes. We also carried out various feature extraction techniques such as Ngram, dependency features, co-relational features and determined the sentiment of generated captions.

Keywords: Image Sentiment analysis; Emotic dataset; CNN; EfficientNetB7; Attention based LSTM; GRU


  1. Chandrasekaran G, Antoanela N, Andrei G, Monica C, Hemanth J. Visual Sentiment Analysis Using Deep Learning Models with Social Media Data. Applied Sciences. 2022;12(3):1030. Available from: https://doi.org/10.3390/app12031030
  2. Hoang MH, Kim SH, Yang HJ, Lee GS. Context-Aware Emotion Recognition Based on Visual Relationship Detection. IEEE Access. 2021;9:90465. Available from: https://doi.org/10.1109/ACCESS.2021.3091169
  3. Wankhade M, Rao ACS, Kulkarni C. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review. 2022;55(7):5731–5780. Available from: https://doi.org/10.1007/s10462-022-10144-1
  4. Meena G, Mohbey KK, Indian A, Kumar S. Sentiment Analysis from Images using VGG19 based Transfer Learning Approach. Procedia Computer Science. 2022;204:411. Available from: https://doi.org/10.1016/j.procs.2022.08.050
  5. Agughalam D, Pathak P, Stynes P. Bidirectional LSTM approach to image captioning with scene features. Thirteenth International Conference on Digital Image Processing (ICDIP 2021). 2021;11878. Available from: https://doi.org/10.1117/12.2600465
  6. Chandrasekaran G, Hemanth DJ. Efficient Visual Sentiment Prediction Approaches Using Deep Learning Models. In: Knowledge Graphs and Semantic Web. (pp. 260-272) Springer International Publishing. 2021.
  7. Huang F, Wei K, Weng J, Li Z. Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis. ACM Transactions on Multimedia Computing, Communications, and Applications. 2020;16(3):1–19. Available from: https://doi.org/10.1145/3388861
  8. Gelli F, Uricchio T, He X, Bimbo AD, Chua TSS. Learning Subjective Attributes of Images from Auxiliary Sources. In: Proceedings of the 27th ACM International Conference on Multimedia. (pp. 2263-2271) ACM. 2019.
  9. Ou H, Qing C, Xu X, Jin J. Multi-Level Context Pyramid Network for Visual Sentiment Analysis. Sensors. 2021;21(6):2136. Available from: https://doi.org/10.3390/s21062136
  10. Zhao T, Hu Y, Valsdottir LR, Zang T, Peng J. Identifying drug–target interactions based on graph convolutional network and deep neural network. Briefings in Bioinformatics. 2021;22(2):2141–2150. Available from: https://doi.org/10.1093/bib/bbaa044
  11. Masood MA, Abbasi RA, Keong NW. Context-Aware Sliding Window for Sentiment Classification. IEEE Access. 2020;8:4870–4884. Available from: https://doi.org/10.1109/ACCESS.2019.2963586
  12. Shilpa PC, Shereen R, Jacob S, Vinod P. Sentiment Analysis Using Deep Learning. Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV). 2021. Available from: https://doi.org/10.1109/ICICV50876.2021.9388382
  13. Singh C, Imam T, Wibowo S, Grandhi S. A Deep Learning Approach for Sentiment Analysis of COVID-19 Reviews. Applied Sciences. 2022;12(8):3709. Available from: https://doi.org/10.3390/app12083709


© 2022 Agarwal & Gupta.  This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)


Subscribe now for latest articles and news.