Research

From Perception to Clinical Reasoning: Disentangled, Evidence-Grounded Multimodal Learning for Robust Medical Image Analysis and Reporting

Status

Proposal defended April 2026 · expected December 2026

Advisor

Dr. Dong Hye Ye

Scope

7 clinical domains · 13 benchmarks · 8 architectures

Reading a medical image involves five judgments: locating a finding, grading it, weighing it against clinical context, comparing it with earlier studies, and committing to a claim. This dissertation builds methods for each stage on two ideas: every stage carries a distribution rather than a point estimate, and evidence from different sources is represented separately so that losing one source does not corrupt the others. It is evaluated across endoscopy, whole-slide histology, retinal photography, brain MRI with genomics, and chest radiography.

1. 01

   Stage 1: Localization

   Where is the evidence?

   • DCAN

     • Dynamic Contextual Attention Network: Transforming Spatial Representations into Adaptive Insights for Endoscopic Polyp Diagnosis · 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
2. 02

   Stage 2: Characterization

   What grade? What co-occurs?

   • CAP-WAE

     • Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading · 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: The Second Workshop on GenAI for Health: Potential, Trust, and Policy Compliance

   • Sparse expert routing

     • Disentangling Co-Occurring Retinal Pathologies with Saliency-Guided Sparse Expert Routing · 36th IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2026)

   • C²A

     • C²A: Coupling Spatial Evidence with Clinical Priors via Co-occurrence Aware Class Attention for Multi-Label Chest X-Ray Classification · 36th IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2026)
3. 03

   Stage 3: Contextualization

   Which source supports the reading, and what if context is missing?

   • DREAM

     • DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation · 48th International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)

   • DiA-gnostic VLVAE

     • DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities · 40th Annual Conference on Association for the Advancement of Artificial Intelligence (AAAI 2026)

   • SC-CoMoE

     • SC-CoMoE: Self-Critiquing Cooperative Mixture-of-Experts for Multi-Modal Mobile Oculomics · Submitted
4. 04

   Stage 4: Comparison

   How has the finding changed since the prior study?

   • Dist-RRG

     • Dist-RRG: Distributional Change Modeling on Statistical Manifolds for Longitudinal Radiology Report Generation · Under review

   • STRIDE

     • STRIDE: Stochastic Trajectory Reasoning via Iterative Distributional Evolution in Longitudinal Radiology Report Generation · Under review
5. 05

   Stage 5: Commitment

   Can the claim be checked against the image?

   • DELTA

     • Claims You Can See: Differential-Evidence Learning of Temporal Assertions for Longitudinal Radiology Report Generation · Under review

Attentive Multi-modal Learning: Unifying Visual and Clinical Insights for Medical Image Classification and Report Generation

Integrates medical images, clinical text and genetics through scalable attention networks and vision-language models. Introduces Spatial Sequence Attention (SSA) for schizophrenia diagnosis from structural MRI, the Multi-modal Imaging Genomics Transformer (MIGTrans) that adds functional connectivity and SNP genomics, and the Multi-Modal Medical Transformer (M3T) for retinal report generation.