Indian Journal of Science and Technology
Year: 2023, Volume: 16, Issue: 34, Pages: 2703-2708
Anupama Navuduri1*, Siddhivinayak Kulkarni1
1School of Computer Engineering and Technology, Dr. Vishwanath Karad MIT World Peace University, Pune, India
Email: [email protected]
Received Date:06 June 2023, Accepted Date:04 August 2023, Published Date:12 September 2023
Objectives: The purpose of this research is to use a multimodal learning approach to perform suggestive diagnosis and generate reports based on chest X-rays and associated data. This research falls under the Vision Language Generation, or VLG, which in this case produces reports given a chest X-ray. Methods: We use a Transformer model with CNN and RNN as part of a multimodal architecture in addition to greedy beam search to generate report impressions in order to construct a proper transformer model capable of producing precise report impressions. We will also collect reports and chest X-rays from the dataset in order to evaluate the adaptability of the model: Indiana University’s Open-I CXR(1). This will be done so that the results can be evaluated and the model’s ability to produce accurate and grammatically correct impressions of reports can be improved. Findings: We achieved better BLEU-1 and BLEU-2 scores compared to the research selected for this research. We have been able to achieve following BLEU scores through our proposed model: BLEU-1 = 0.592, BLEU-2 = 0.422, BLEU-3 = 0.298, BLEU-4 = 0.205. Novelty: We propose a Transformer model to generate report impressions. This transformer model has CNN as an encoder and RNN as a decoder with attention mechanism on top of it. Additionally, greedy beam search has been used to get grammatically correct sentences.
Keywords: Chest XRay; Transformers; OpenI CXR; CNN; RNN
2023 Navuduri & Kulkarni. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Published By Indian Society for Education and Environment (iSee)
Subscribe now for latest articles and news.