Indian Journal of Science and Technology

Abstract

Objectives: The purpose of this research is to use a multimodal learning approach to perform suggestive diagnosis and generate reports based on chest X-rays and associated data. This research falls under the Vision Language Generation, or VLG, which in this case produces reports given a chest X-ray. Methods: We use a Transformer model with CNN and RNN as part of a multimodal architecture in addition to greedy beam search to generate report impressions in order to construct a proper transformer model capable of producing precise report impressions. We will also collect reports and chest X-rays from the dataset in order to evaluate the adaptability of the model: Indiana University’s Open-I CXR(1). This will be done so that the results can be evaluated and the model’s ability to produce accurate and grammatically correct impressions of reports can be improved. Findings: We achieved better BLEU-1 and BLEU-2 scores compared to the research selected for this research. We have been able to achieve following BLEU scores through our proposed model: BLEU-1 = 0.592, BLEU-2 = 0.422, BLEU-3 = 0.298, BLEU-4 = 0.205. Novelty: We propose a Transformer model to generate report impressions. This transformer model has CNN as an encoder and RNN as a decoder with attention mechanism on top of it. Additionally, greedy beam search has been used to get grammatically correct sentences.

Keywords: Chest XRay; Transformers; OpenI CXR; CNN; RNN