• P-ISSN 0974-6846 E-ISSN 0974-5645

Indian Journal of Science and Technology


Indian Journal of Science and Technology

Year: 2023, Volume: 16, Issue: 30, Pages: 2304-2310

Original Article

Multimodal Approach to Recommend Movie Genres Based on Multi Datasets

Received Date:22 May 2023, Accepted Date:06 July 2023, Published Date:08 August 2023


Objectives: The main purpose of this study is to implement the multimodal as well as the multilabel approach of accurate movie genre classification using the Actor, Director, Writer, Poster Images, and Synopsis data. The various kind of data collected from IMDB as well as existing datasets are used to train the models. Methods: The 5 deep learning models are created on Poster Images, Text Data, Actors Data, Directors Data, and Writers Data. These models are then combined using the weighted average ensemble model of deep learning. The final output gives the prediction scores of the top 3 genres for a given movie. The main deep learning model used for poster image model training is CNN. The LSTM model is used for text/synopsis model training. The multilabel along with the multimodal approach yielded good results in terms of accurate predictions than the existing models. Findings: The total F1 Score is 0.65. Along with good results, it has some limitations which are further discussed in the paper. Novelty: The novelty of the work lies in 3 major aspects, firstly the consideration of the Actor, Director, and Writer data for the genre classification, as usually, an actor does a particular kind of movie, a writer writes a particular type of movie and director directs a particular type or genre movie. Although there are many exceptions but usually this is the case. The second novelty lies in combining the models using the weighted average method wherein every model is assigned a weight for how much it will contribute to the final genre classification. And lastly, the novelty lies in how the weights are defined i.e., we used the correlation method to determine the weights of every individual model for genre classification.

Keywords: Movie Genre Classification; CNN; Multimodal Movie Genre; Multilabel Movie Genre; LSTM; Deep Learning


  1. Agarwal A, Das RR, Das AR. Machine Learning Techniques for Automated Movie Genre Classification Tool. 2021 4th International Conference on Recent Developments in Control, Automation & Power Engineering (RDCAPE). 2021. Available from: https://doi.org/10.1109/RDCAPE52977.2021.9633422
  2. Hossain N, Ahamad MM, Aktar S, Moni MA. Movie Genre Classification with Deep Neural Network using Poster Images. 2021 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD). 2021. Available from: https://doi.org/10.1109/icict4sd50815.2021.9396778
  3. Braz L, Teixeira V, Pedrini H, Dias Z. Image-Text Integration Using a Multimodal Fusion Network Module for Movie Genre Classification. 11th International Conference of Pattern Recognition Systems (ICPRS 2021). 2021. Available from: https://doi.org/10.1049/icp.2021.1456
  4. Bi T, Jarnikov D, Lukkien J. Video Representation Fusion Network For Multi-Label Movie Genre Classification. 2020 25th International Conference on Pattern Recognition (ICPR). 2021. Available from: https://doi.org/10.1109/icpr48806.2021.9412480
  5. Mangolin RB, Pereira RM, Britto AS, Silla CN, Feltrim VD, Bertolini D. A multimodal approach for multi-label movie genre classification. 2020. Available from: https://doi.org/10.1007/s11042-020-10086-2
  6. Nadem F, Mahdian R, Zareian H. Genre Classification of Movies from a Single Poster Image Using Feature Fusion. 2021 7th International Conference on Signal Processing and Intelligent Systems (ICSPIS). 2021. Available from: https://doi.org/10.1109/icspis54653.2021.9729380
  7. Paulino MAD, Costa YMG, Feltrim VD. Evaluating multimodal strategies for multi-label movie genre classification. 2022 29th International Conference on Systems, Signals and Image Processing (IWSSIP). 2022. Available from: https://doi.org/10.1109/iwssip55020.2022.9854451
  8. Akbar J, Utami E, Yaqin A. Multi-Label Classification of Film Genres Based on Synopsis Using Support Vector Machine, Logistic Regression and Naïve Bayes Algorithms. 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE). 2022. Available from: https://doi.org/10.1109/icitisee57756.2022.10057828


© 2023 Khant & Tidke. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)


Subscribe now for latest articles and news.