Generating diagnostic report for medical image by high-middle-level visual information incorporation on double deep learning models.
BACKGROUND AND OBJECTIVES
Writing diagnostic reports for medical images is a heavy and tedious work. The automatic generation of medical image diagnostic reports can assist doctors to reduce their workload and improve diagnosis efficiency. It is of great significance to introduce image caption algorithm into medical image processing. Existing approaches attempt to generate medical image diagnostic reports using image caption algorithms but without taking the accuracy of pathological information in generated diagnostic reports into account.


METHODS
To solve the mentioned problem, we propose a Semantic Fusion Network (SFNet) including a lesion area detection model and a diagnostic generation model. The lesion area detection model can extract visual and pathological information from medical image, and the diagnostic report generation model can learn to fuse the two kinds of information to generate reports. Thus, the pathological information in the generated diagnostic reports can be more accurate.


RESULTS
Experimental results have verified the performance of our model (Accuracy increases 1.2% on the Ultrasound Image Dataset and 2.4% on the Open-i X-ray Image Dataset), compared with the model only using visual feature to generate diagnostic reports.


CONCLUSIONS
This work utilizes computer algorithms to generate the more accurate diagnostic reports for medical images automatically, which expands the application of computer-aided diagnosis and promotes the implementation of deep learning in the medical image analysis field.
['Zeng|Xianhua|X|', 'Wen|Li|L|', 'Xu|Yang|Y|', 'Ji|Conghui|C|']
D000465:Algorithms D000077321:Deep Learning D003936:Diagnosis, Computer-Assisted D007091:Image Processing, Computer-Assisted D014463:Ultrasonography
Medical image Image caption Pathological information Computer-aided diagnosis Object detection