This paper presents a novel deep learning-based approach named RealDiffFusionNet incorporating Neural Controlled Differential Equations (Neural CDE) - time series models that are robust in handling irregularly sampled data - and multi-head attention to align relevant multimodal context (image data, time invariant data, etc.) at each time point. Long short-term memory (LSTM) models were also used as a baseline. Two different datasets were used: a data from the Open-Source Imaging Consortium (OSIC) containing structured time series data of demographics and lung function with a baseline CT scan of the lungs and the second from the Alzheimer's Disease Neuroimaging Initiative (ADNI) containing a series of MRI scans along with demographics, physical examinations, and cognitive assessment data. An ablation study was performed to understand the role of CDEs, multimodal data, attention fusion, and interpolation strategies on model performance. When the baseline models were evaluated, the use of multimodal data resulted in an improvement in Neural CDE performance, with a lower test RMSE. Additionally, the performance of multimodal Neural CDE was also superior to multimodal LSTM. In the attention-based architectures, fusion through concatenation and rectilinear interpolation were found to improve model performance. The performance of the proposed RealDiffFusionNet was found to be superior (0.2570) to all models. For the ADNI dataset, between the Neural-CDE and LSTM models trained only on the structured data, the test RMSE were comparable (0.471 for LSTM vs. 0.4581 Neural-CDE). Furthermore, the addition of image features from patients' MRI series resulted in an improvement in performance, with a lower test RMSE (0.4372 with multimodal vs 0.4581 with structured data). RealDiffFusionNet has shown promise in utilizing CDEs and multimodal data to accurately predict disease progression.